Skip to content

[Compiler]: Using Located<T> instead of std::pair<SourceLoc, T> #28643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

kitaisreal
Copy link
Contributor

@kitaisreal kitaisreal commented Dec 8, 2019

  1. Created Located struct in swift/Basic
  2. Updated usages throughout the compiler

https://bugs.swift.org/browse/SR-11889

@varungandhi-apple

friend bool operator ==(const Located<U> lhs, const Located<U> rhs) {
return lhs.item == rhs.item && lhs.loc == rhs.loc;
}
};
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add a dump method that does something reasonable like for other types? You could have two overloads: void dump() const; void dump(llvm::raw_ostream &OS) const where the former calls the latter using llvm::errs(). [For example, SourceFile does this.]

You could add a new file Located.cpp which has the implementations and the appropriate #includes. You will also need to add Located.cpp to the lib/Basic/CMakeLists.txt file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kitaisreal what do you think about this point? I think it might be useful to have this method for debugging.

As to what is should print, perhaps something that prints the SourceLoc first and then the item, leaving a space in between (say)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@varungandhi-apple I agree, that we should introduce such function. Added.

@varungandhi-apple
Copy link
Contributor

varungandhi-apple commented Dec 9, 2019

Thanks for the PR! I've made some suggestions for changes. The == overload is fine, I didn't think of it when writing the issue. Could you run a string search for std::pair<SourceLoc, (maybe you already did this and I missed it)? I think there are some places which have the type parameters the other way around and it would be to replace those usage sites too.

@swift-ci please smoke test

@kitaisreal
Copy link
Contributor Author

Updated for pairs like std::pair<SourceLoc, ...>

@varungandhi-apple
Copy link
Contributor

@swift-ci please smoke test

@varungandhi-apple
Copy link
Contributor

varungandhi-apple commented Dec 10, 2019

Sorry, I didn't pick this up earlier, but I think we should have the field names in PascalCase, so Item and Loc, for consistency with the rest of the codebase.

@kitaisreal
Copy link
Contributor Author

@varungandhi-apple Changed Located field names in PascalCase, Item and Loc.

@varungandhi-apple
Copy link
Contributor

@kitaisreal there are some merge conflicts. So you might want to rebase on top of latest master, fixing that conflict.

When you rebase, you'll pick up a patch I got merged recently (commit: fb59293), which introduces a struct Convention -- that has two fields StringRef ClangType and SourceLoc ClangTypeLoc. It would be better to represent it as one field Located<StringRef> ClangType instead and change downstream code appropriately. If you could adjust that as well, that'd be awesome. But yeah, not strictly necessary for the scope of your changes -- if you don't feel like doing it, feel free to skip that 😄.

@kitaisreal kitaisreal force-pushed the using-located-instead-of-pair branch from 45ba939 to e89e2dc Compare December 15, 2019 17:43
@kitaisreal
Copy link
Contributor Author

@varungandhi-apple fixed conflicts, also updated Convention to use Located.

@varungandhi-apple
Copy link
Contributor

Awesome. I'm traveling for the next ~2 days, I'll get back to reviewing after that.

@swift-ci please smoke test

@kitaisreal
Copy link
Contributor Author

@varungandhi-apple currently the build from utils/build-script is passing without any errors. But on CI we are trying to build a toolchain and it is failing in lldb ? Can I reproduce such build error without trying to build a toolchain ?

@varungandhi-apple
Copy link
Contributor

You can use build-script to build LLDB as well. Here's one example:

./swift/utils/build-script \
  --lldb \
  --release-debuginfo \
  --test \
  -- \
  --skip-build-benchmarks \
  --lldb-use-system-debugserver \
  --skip-test-swift

/// The original source location from which the item was parsed.
SourceLoc Loc;

Located(): Item(), Loc() {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, I wonder if it makes sense to have this constructor given that not every T can be default or value-initialized. It might make more sense to provide an explicit constructor from a T or const T&, or default the loc in the other constructor and make it explicit.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this is the route you take, please also define an explicit move constructor.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the reason for adding this was that DenseMap required this constructor for its values?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should give a DenseMapInfo specialization to avoid that...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @CodaFi
As I understand correctly from code base we are making such specializations for Key:

namespace llvm {

template <typename T> struct DenseMapInfo;

template<typename T>
struct DenseMapInfo<swift::Located<T>> {

  static inline swift::Located<T> getEmptyKey() {
    return swift::Located<T>(T(), swift::SourceLoc());
  }

  static inline swift::Located<T> getTombstoneKey() {
    return swift::Located<T>(T(), swift::SourceLoc());
  }

  static unsigned getHashValue(const swift::Located<T> &LocatedVal) {
    return combineHashValue(DenseMapInfo<T>::getHashValue(LocatedVal.Item),
                            DenseMapInfo<swift::SourceLoc>::getHashValue(LocatedVal.Loc));
  }

  static bool isEqual(const swift::Located<T> &LHS, const swift::Located<T> &RHS) {
    return LHS == RHS;
  }
};
} // namespace llvm

But problem in DenseMap

value_type& FindAndConstruct(const KeyT &Key) {
    BucketT *TheBucket;
    if (LookupBucketFor(Key, TheBucket))
      return *TheBucket;

    return *InsertIntoBucket(TheBucket, Key);
  }

  ValueT &operator[](const KeyT &Key) {
    return FindAndConstruct(Key).second /**here**/;
  }

As I understand it tries to default initialize Value. And solution with DenseMapInfo specialization will not help in such case, where Located is Value. Is it right ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tombstone and empty key requirements are not correctly implemented. Take a look at how LLVM implements this for std::pair to get an idea of how this should look. You need to delegate to the DenseMapInfo of T and the SourceLoc.

template<typename U>
friend bool operator ==(const Located<U> lhs, const Located<U> rhs) {
friend bool operator ==(const Located<U>& lhs, const Located<U>& rhs) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CodaFi's comment was probably about this line, which should have const Located<U> &lhs instead of const Located<U>& lhs and similarly for rhs.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@varungandhi-apple
Copy link
Contributor

Mostly LGTM but you'll need to resolve the merge conflicts and address the failing LLDB compilation. Feel free to @ me or Robert (CodaFi) to run the tests.

@CodaFi
Copy link
Contributor

CodaFi commented Dec 23, 2019

@swift-ci please test

1 similar comment
@CodaFi
Copy link
Contributor

CodaFi commented Dec 23, 2019

@swift-ci please test

@swift-ci
Copy link
Contributor

Build failed
Swift Test OS X Platform
Git Sha - 646e9ac

@swift-ci
Copy link
Contributor

Build failed
Swift Test Linux Platform
Git Sha - 646e9ac

@kitaisreal
Copy link
Contributor Author

kitaisreal commented Dec 26, 2019

Hi, @CodaFi @varungandhi-apple I was able to successfully build lldb locally, but small changes were introduced in this repository https://github.com/apple/llvm-project.
Should I create pull request with the same name in https://github.com/apple/llvm-project ? How will CI test two branches in two repositories together ?

@kitaisreal kitaisreal requested a review from CodaFi December 26, 2019 21:00
@CodaFi
Copy link
Contributor

CodaFi commented Dec 26, 2019

Branch from swift/master and submit your changes to Apple/llvm-project targeting swift/master. We can trigger CI with paired pull requests and manually merge them as well once the holiday lock is over.

@kitaisreal
Copy link
Contributor Author

kitaisreal commented Dec 27, 2019

@CodaFi, @varungandhi-apple created pull request swiftlang/llvm-project#522.

@varungandhi-apple
Copy link
Contributor

swiftlang/llvm-project#522

@swift-ci please test

@swift-ci
Copy link
Contributor

swift-ci commented Jan 3, 2020

Build failed
Swift Test OS X Platform
Git Sha - 646e9ac

@swift-ci
Copy link
Contributor

swift-ci commented Jan 3, 2020

Build failed
Swift Test Linux Platform
Git Sha - 646e9ac

@varungandhi-apple
Copy link
Contributor

@kitaisreal can you look at the test failures? It looks like DiagnosticConsumerTests.cpp is missing a #include "swift/Basic/Located.h".

/Users/buildnode/jenkins/workspace/swift-PR-osx/branch-master/swift/unittests/AST/DiagnosticConsumerTests.cpp:20:30: error: no template named 'Located'
  using ExpectedDiagnostic = Located<StringRef>;
                             ^

@kitaisreal
Copy link
Contributor Author

@varungandhi-apple Sure. Updated.

@varungandhi-apple
Copy link
Contributor

swiftlang/llvm-project#522

@swift-ci please test

@swift-ci
Copy link
Contributor

swift-ci commented Jan 6, 2020

Build failed
Swift Test Linux Platform
Git Sha - 646e9ac

@swift-ci
Copy link
Contributor

swift-ci commented Jan 6, 2020

Build failed
Swift Test OS X Platform
Git Sha - 646e9ac

@varungandhi-apple varungandhi-apple merged commit 022314a into swiftlang:master Jan 6, 2020
@compnerd
Copy link
Member

compnerd commented Jan 6, 2020

@compnerd
Copy link
Member

compnerd commented Jan 6, 2020

@varungandhi-apple
Copy link
Contributor

@compnerd yes, this is related. I merged it after the tests were passing for Linux and macOS. Is this an MSVC bug? Don't understand why it is complaining about a duplicate, there is exactly one operator == present.

@compnerd
Copy link
Member

compnerd commented Jan 6, 2020

I dion't think that this is a bug at all. I think that the compiler is identifying an issue with the code. The warning is documented here https://docs.microsoft.com/en-us/cpp/error-messages/compiler-errors-2/compiler-error-c2995?view=vs-2019 . The template is getting instantiated multiple times. I think I have a fix for this, I'll push up a change.

@compnerd
Copy link
Member

compnerd commented Jan 6, 2020

See #29032

@varungandhi-apple
Copy link
Contributor

I tried compiling this locally and I saw the same error with Clang. Wondering how it got through in CI. 😕

@davezarzycki
Copy link
Contributor

davezarzycki commented Jan 7, 2020

This fails to build on Fedora 31 (x86_64) and I'm surprised this builds at all. I can't find a declaration or definition of operator<<(llvm::raw_ostream &, const swift::SourceLoc &) anywhere. How does this compile? Can we revert this? Or add the output operator? Or remove the offending line?

/home/dave/s/u/swift/lib/Basic/Located.cpp:23:6: error: invalid operands to binary expression ('llvm::raw_ostream' and 'const swift::SourceLoc')
  os << Loc << " " << Item;
  ~~ ^  ~~~
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:202:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const void *' for 1st argument; take the address of the argument with &
  raw_ostream &operator<<(const void *P);
               ^
/home/dave/s/u/llvm/include/llvm/ADT/Optional.h:431:14: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'llvm::NoneType' for 2nd argument
raw_ostream &operator<<(raw_ostream &OS, NoneType);
             ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:146:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'char' for 1st argument
  raw_ostream &operator<<(char C) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:153:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'unsigned char' for 1st argument
  raw_ostream &operator<<(unsigned char C) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:160:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'signed char' for 1st argument
  raw_ostream &operator<<(signed char C) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:167:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'llvm::StringRef' for 1st argument
  raw_ostream &operator<<(StringRef Str) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:182:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const char *' for 1st argument
  raw_ostream &operator<<(const char *Str) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:189:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const std::string' (aka 'const basic_string<char, char_traits<char>, allocator<char> >') for 1st argument
  raw_ostream &operator<<(const std::string &Str) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:194:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const SmallVectorImpl<char>' for 1st argument
  raw_ostream &operator<<(const SmallVectorImpl<char> &Str) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:198:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'unsigned long' for 1st argument
  raw_ostream &operator<<(unsigned long N);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:199:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'long' for 1st argument
  raw_ostream &operator<<(long N);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:200:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'unsigned long long' for 1st argument
  raw_ostream &operator<<(unsigned long long N);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:201:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'long long' for 1st argument
  raw_ostream &operator<<(long long N);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:204:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'unsigned int' for 1st argument
  raw_ostream &operator<<(unsigned int N) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:208:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'int' for 1st argument
  raw_ostream &operator<<(int N) {
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:212:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'double' for 1st argument
  raw_ostream &operator<<(double N);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:229:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const llvm::format_object_base' for 1st argument
  raw_ostream &operator<<(const format_object_base &Fmt);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:232:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const llvm::FormattedString' for 1st argument
  raw_ostream &operator<<(const FormattedString &);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:235:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const llvm::FormattedNumber' for 1st argument
  raw_ostream &operator<<(const FormattedNumber &);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:238:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const llvm::formatv_object_base' for 1st argument
  raw_ostream &operator<<(const formatv_object_base &);
               ^
/home/dave/s/u/llvm/include/llvm/Support/raw_ostream.h:241:16: note: candidate function not viable: no known conversion from 'const swift::SourceLoc' to 'const llvm::FormattedBytes' for 1st argument
  raw_ostream &operator<<(const FormattedBytes &);
               ^
/home/dave/s/u/llvm/include/llvm/ADT/Optional.h:435:14: note: candidate template ignored: could not match 'Optional<type-parameter-0-0>' against 'const swift::SourceLoc'
raw_ostream &operator<<(raw_ostream &OS, const Optional<T> &O) {
             ^
1 error generated.

@davezarzycki
Copy link
Contributor

davezarzycki commented Jan 7, 2020

There are two fundamental problems here:

  1. This code doesn't compile with top-of-tree clang, which is more reliable about emitting templated functions that are marked "used", i.e. what SWIFT_DEBUG_DUMP does among other things. This same bug was tripped over when the SWIFT_DEBUG_DUMP macro was introduced.
  2. operator<< on a SourceLoc is arguably meaningless. One also needs a SourceManager reference and a buffer ID in order to extract the line and column information.

I'm just going to comment the problematic line out, since older clangs don't emit the function anyway and @varungandhi-apple seemingly did not notice. See #29045

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants