Compile from Tasty #507

odersky · 2015-04-28T10:11:40Z

New compiler main module: FromTasty.

To be able able to post process pickled TASTY tree we let ClassfileParser#run return any embedded unpickler. The unpickler is further propagated by a new method ClassFileLoader#load in SymbolLoaders.

`FromTasty` is a main module which compiles TASTY info instead of sources.

Previously, the empty package was always initialized with the empty scope. This means that separate compilation between files in the empty package was not possible.

So far: Only one source file is recorded. Should evaluate whether more are needed. Will programs composed from several source files be pickled? They will certainly be generated after inlining, but maybe all that happens after pickling?

A missing delta position signifies that the node has the same position as its parent. Once that case was added, we can now enable positions when reading from Tasty.

DarkDimius · 2015-04-28T10:23:45Z

@odersky is it ready to be tested?
If yes, can you please add several tests?

odersky · 2015-04-28T10:27:25Z

That depends whether macros are expanded before or after pickling. Either is possible IMO.

odersky · 2015-04-28T10:28:55Z

Currently, we can only test classes, not objects. The problem is that the backend puts the TASTY information in the wrong file. For a top-level object like Labels, it should be put in Labels.class, but it is put in Labels$.class. @DarkDimius can you add a commit for this?

xeno-by · 2015-04-28T10:29:11Z

@DarkDimius What would you prefer? Before or after pickling? Expanding before would allow to report errors early.

DarkDimius · 2015-04-28T10:31:34Z

@DarkDimius can you add a commit for this?

Ok, I will.

odersky · 2015-04-28T10:33:21Z

Usage information is in FromTasty. I have not got around to write test infrastructure for this. Essentially you need to compile first a Scala file, then call FromTasty with the output directory on the classpath.

Currently tested that this works: In directory pickling (where classes needs to be on the classpath):

dotc -d classes Coder
java dotty.tools.dotc.FromTasty Coder

succeeds with 6 warnings.

dotc extmethods.scala
java dotty.tools.dotc.FromTasty extMethods.T

succeeds.

@VladimirNik Can you try some other files and add tests to tests.scala?

odersky · 2015-04-28T10:46:15Z

@xeno-by Early or late is actually not so much of a concern. The normal compiler will anyway immediately proceed after pickling, so expansion will happen anyway then. It's more a question
of not having to redo expansions in (non-inline) library modules.

And I think that settles it: We do not want to be forced to touch trees of dependencies except for their top-level symbols. the linker will read them, but that should be optional. So We will need to cope with multi-source units.

DarkDimius · 2015-04-28T10:47:23Z

Before or after pickling? Expanding before would allow to report errors early.

That's a hard question. As inline redoes overload resolution, than inline has new particularities with separate compilation. In order to have more inline methods discovered, we would want to perform inline expansion as late as possible.
But from linker perspective, if macros or inline are not expanded - it cannot save any information about the trees, and will need to read whole world.

These only exist if there was a pickler, and they are not unique per CompilationUnit.

xeno-by · 2015-04-28T11:01:19Z

Actually, that's something that I wanted to discuss. Can we revisit the point of inline redoing overload resolution? I'm afraid that it might cause problems later on and this is one of the examples. @odersky, are we open to alternatives?

VladimirNik · 2015-04-29T00:48:43Z

@odersky Yes, I'll try other tests.

VladimirNik · 2015-04-30T02:03:16Z

I tried to run FromTasty with Coder and extmethods, and in my environment it fails for both of the test files.

I tried to use FromTasty with simplified example (based on Coder) and it works:

class mytest {
  private val mnemonics = Map('2' -> "ABC")
  val str = "abc"
}

If modify example:

class mytest {
  private val mnemonics = Map('2' -> "ABC")
  val str = "abc"
  for ((digit, str) <- mnemonics; ltr <- str ) {}
}

it fails with:

error lambda lifting Simple(mytest.scala): value x22 is not visible from <none>
error lambda lifting Simple(mytest.scala): value x22 is not visible from class mytest
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2()
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2().asInstanceOf
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2().asInstanceOf[String]
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing val str: String = x22._2().asInstanceOf[String]
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing {
  val str: String = x22._2().asInstanceOf[String]
  {
    augmentString(str).foreach({
      def $anonfun(ltr: Char): Unit = {
        ()
      }
      closure($anonfun)
    })
  }
}
...
...
Exception in thread "main" dotty.tools.dotc.transform.LambdaLift$NoPath
    at dotty.tools.dotc.transform.LambdaLift$LambdaLifter.dotty$tools$dotc$transform$LambdaLift$LambdaLifter$$markFree(LambdaLift.scala:148)
    at dotty.tools.dotc.transform.LambdaLift$LambdaLifter.dotty$tools$dotc$transform$LambdaLift$LambdaLifter$$markFree(LambdaLift.scala:155)
    at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:204)
    at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:827)
    at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:820)
    at dotty.tools.dotc.ast.Trees$Instance$TreeAccumulator.foldOver(Trees.scala:1198)
    at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:221)
    at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:827)
    at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:820)
    at dotty.tools.dotc.ast.Trees$Instance$TreeAccumulator.foldOver(Trees.scala:1204)
    at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:221)

In this case it seems that the problem is with the usage of str inside for.

Also it seems that FromTasty modifies .class files (remove TASTY attribute). If try to run FromTasty second time on the same class (that was successfully pickled/unpickled), there is an error:

error: class mytest cannot be unpickled because its class file ./mytest.class does not have a TASTY attribute.

odersky · 2015-04-30T06:39:38Z

Did you run with -Ycheck:frontend? That would likely tell you of errors
before they reach pickling.

On Thu, Apr 30, 2015 at 4:03 AM, Vladimir Nikolaev <[email protected]

wrote:

I tried to run FromTasty with Coder and extmethods, and in my environment
it fails for both of the test files.

I tried to use FromTasty with simplified example (based on Coder) and it
works:

class mytest {
private val mnemonics = Map('2' -> "ABC")
val str = "abc"
}

If modify example:

class mytest {
private val mnemonics = Map('2' -> "ABC")
val str = "abc"
for ((digit, str) <- mnemonics; ltr <- str ) {}
}

it fails with:

error lambda lifting Simple(mytest.scala): value x22 is not visible from
error lambda lifting Simple(mytest.scala): value x22 is not visible from class mytest
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2()
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2().asInstanceOf
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing x22._2().asInstanceOf[String]
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing val str: String = x22._2().asInstanceOf[String]
dotty.tools.dotc.transform.LambdaLift$NoPath while traversing {
val str: String = x22._2().asInstanceOf[String]
{
augmentString(str).foreach({
def $anonfun(ltr: Char): Unit = {
()
}
closure($anonfun)
})
}
}
...
...
Exception in thread "main" dotty.tools.dotc.transform.LambdaLift$NoPath
at dotty.tools.dotc.transform.LambdaLift$LambdaLifter.dotty$tools$dotc$transform$LambdaLift$LambdaLifter$$markFree(LambdaLift.scala:148)
at dotty.tools.dotc.transform.LambdaLift$LambdaLifter.dotty$tools$dotc$transform$LambdaLift$LambdaLifter$$markFree(LambdaLift.scala:155)
at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:204)
at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:827)
at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:820)
at dotty.tools.dotc.ast.Trees$Instance$TreeAccumulator.foldOver(Trees.scala:1198)
at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:221)
at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:827)
at dotty.tools.dotc.ast.tpd$EnclosingMethodTraverser.apply(tpd.scala:820)
at dotty.tools.dotc.ast.Trees$Instance$TreeAccumulator.foldOver(Trees.scala:1204)
at dotty.tools.dotc.transform.LambdaLift$LambdaLifter$CollectDependencies.traverse(LambdaLift.scala:221)

In this case it seems that the problem is with the usage of str inside fo.

Also it seems that FromTasty modifies .class files (remove TASTY
attribute). If try to run FromTasty second time on the same class (that was
successfully pickled/unpickled), there is an error:

error: class mytest cannot be unpickled because its class file ./mytest.class does not have a TASTY attribute.

—
Reply to this email directly or view it on GitHub
#507 (comment).

Martin Odersky
EPFL

DarkDimius · 2015-04-30T09:17:37Z

I'd recommend to run with -Ycheck:all.

VladimirNik · 2015-04-30T11:46:32Z

@odersky , @DarkDimius With these options there are no errors.

Other issue is that after the first run of FromTasty, the same, previously correct .class file can't be unpickled during the second run of FromTasty:

error: class mytest cannot be unpickled because its class file ./mytest.class does not have a TASTY attribute.

odersky · 2015-04-30T12:58:29Z

@odersky , @DarkDimius With these options there are no errors.

Wow. Need to investigate.

Other issue is that after the first run of FromTasty, the same, previously correct .class file can't be
unpickled during the second run of FromTasty:

That will be hard to fix. I recommend to run FromTasty with an output directory that does not override the classpath. Let's also add a comment to FromTasty's docs that clarifies this point.

We want to ensure that the whole tree is read at phase frontend. To achieve this, we run an empty traverser over the tree.

odersky · 2015-04-30T13:53:48Z

@VladimirNik The problem with the crash in lambdalift should be fixed now.

odersky · 2015-05-02T16:47:32Z

Should we merge this? It will help @VladimirNik progress.

Compile from Tasty

odersky added 7 commits April 28, 2015 12:07

Rename Unpickler -> Scala2Unpickler

36dbbfa

ClassfileParser and SymbolLoader should return embedded unpicklers

dad7999

To be able able to post process pickled TASTY tree we let ClassfileParser#run return any embedded unpickler. The unpickler is further propagated by a new method ClassFileLoader#load in SymbolLoaders.

Compiler for embedded TASTY info.

12f2ee7

`FromTasty` is a main module which compiles TASTY info instead of sources.

Fix definition of empty package so that it can persist members

a4d86e9

Previously, the empty package was always initialized with the empty scope. This means that separate compilation between files in the empty package was not possible.

Unpickler should not label parameter accessors Deferred.

512689c

Maintain source files in pickled info

014ad9b

So far: Only one source file is recorded. Should evaluate whether more are needed. Will programs composed from several source files be pickled? They will certainly be generated after inlining, but maybe all that happens after pickling?

Handle missing delta positions when unpickling

12d5dbc

A missing delta position signifies that the node has the same position as its parent. Once that case was added, we can now enable positions when reading from Tasty.

Add usage documentation to FromTasty

a347632

Move addrOfTree, addOfSym from CompilationUnit to Pickler.

9e2dd75

These only exist if there was a pickler, and they are not unique per CompilationUnit.

Force TASTY trees read by unpickler in FromTasty

10a5f91

We want to ensure that the whole tree is read at phase frontend. To achieve this, we run an empty traverser over the tree.

DarkDimius added a commit that referenced this pull request May 2, 2015

Merge pull request #507 from dotty-staging/add/from-tasty

ac46a0e

Compile from Tasty

DarkDimius merged commit ac46a0e into scala:master May 2, 2015

allanrenucci deleted the add/from-tasty branch December 14, 2017 16:57

Compile from Tasty #507

Compile from Tasty #507

Uh oh!

Conversation

odersky commented Apr 28, 2015

Uh oh!

DarkDimius commented Apr 28, 2015

Uh oh!

odersky commented Apr 28, 2015

Uh oh!

odersky commented Apr 28, 2015

Uh oh!

xeno-by commented Apr 28, 2015

Uh oh!

DarkDimius commented Apr 28, 2015

Uh oh!

odersky commented Apr 28, 2015

Uh oh!

odersky commented Apr 28, 2015

Uh oh!

DarkDimius commented Apr 28, 2015

Uh oh!

xeno-by commented Apr 28, 2015

Uh oh!

VladimirNik commented Apr 29, 2015

Uh oh!

VladimirNik commented Apr 30, 2015

Uh oh!

odersky commented Apr 30, 2015

Uh oh!

DarkDimius commented Apr 30, 2015

Uh oh!

VladimirNik commented Apr 30, 2015

Uh oh!

odersky commented Apr 30, 2015

Uh oh!

odersky commented Apr 30, 2015

Uh oh!

odersky commented May 2, 2015

Uh oh!

Uh oh!