Migrate scala3doc codebase into dotty

.github/workflows/ci.yaml

@@ -2,7 +2,11 @@ name: Dotty CI
 
 on:
   push:
+    paths-ignore:
+      - 'scala3doc/**' # Do not run everything on changes only in docs
   pull_request:
+    paths-ignore:
+      - 'scala3doc/**' # Do not run everything on changes only in docs
   schedule:
     - cron: '0 3 * * *'  # Every day at 3 AM
 

.github/workflows/scala3doc.yaml

@@ -0,0 +1,44 @@
+name: scala3doc CI
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+jobs:
+
+  build:
+    runs-on: [self-hosted, Linux]
+    container: lampepfl/dotty:2020-04-24
+
+    # timeout-minutes: 20 We need much more for first run
+
+    steps:
+      - name: Set JDK 11 as default
+        run: echo "/usr/lib/jvm/java-11-openjdk-amd64/bin" >> $GITHUB_PATH
+
+      - uses: actions/checkout@v2
+      - run: git fetch --prune --unshallow --tags
+
+      - name: Cache Coursier
+        uses: actions/cache@v1
+        with:
+          path: ~/.cache/coursier
+          key: sbt-coursier-cache
+
+      - name: Cache SBT
+        uses: actions/cache@v1
+        with:
+          path: ~/.sbt
+          key: sbt-${{ hashFiles('**/build.sbt') }}
+
+      - name: Compile and test
+        run: cd scala3doc && sbt test
+
+      - name: Locally publish self
+        run: cd scala3doc && sbt publishLocal
+
+      - name: Generate test documentation
+        run: cd scala3doc && sbt generateSelfDocumentation
+
+      - name: Generate documentation with SBT plugin
+        run: cd scala3doc && sbt example-project/doc

scala3doc/.gitignore

@@ -0,0 +1,40 @@
+*.class
+*.log
+libs/
+
+# sbt specific
+.cache
+.history
+.lib/
+dist/*
+target/
+lib_managed/
+src_managed/
+project/boot/
+project/plugins/project/
+
+# Scala-IDE specific
+.scala_dependencies
+.worksheet
+.bsp
+
+# Intellij
+.idea
+
+#integration tests dir
+/test-ws
+
+# to test deploy
+private-*
+repo
+
+.metals
+metals.sbt
+.bloop
+.dotty-ide*
+
+# custom things
+output
+
+.vscode
+

scala3doc/.scalafmt.conf

@@ -0,0 +1 @@
+version = "2.6.3"

scala3doc/README.md

@@ -0,0 +1,164 @@
+# Scala3doc
+
+Scala3doc (name subject to change) is the documentation tool for
+[Dotty](https://github.com/lampepfl/dotty), which is scheduled to become
+Scala 3. It's based on [Dokka](https://github.com/Kotlin/dokka), the
+documentation tool for Kotlin. It uses Tasty-Reflect to access definitions,
+which is an officially supported way to access Dotty's perspective of a
+codebase.
+
+We're aiming to support all the features Scaladoc did, plus new and exciting ones such as:
+
+- Markdown syntax!
+- displaying project and API documentation together on one site!
+- and more!
+
+## Running the project
+
+Use the following commands to generate documentation for this project and for Dotty, respectively:
+
+```
+sbt generateSelfDocumentation
+sbt generateDottyLibDocumentation
+```
+
+To actually view the documentation, the easiest way is to run the following in project root:
+
+```
+cd output
+python3 -m http.server 8080
+```
+
+And afterwards point your browser to `http://localhost:8080/self` or
+`http://localhost:8080/stdLib` for this project and for Dotty documentation
+respectively.
+
+It's not strictly necessary to go through an HTTP server, but because of CORS
+the documentation won't work completely if you don't.
+
+## Developing
+
+At least two of our contributors use [Metals](https://scalameta.org/metals/) to
+work on the project.
+
+For every PR, we build documentation for Scala3doc and Dotty. For example, for
+PR 123 you can find them at:
+
++ https://scala3doc.s3.eu-central-1.amazonaws.com/pr-123/self/main/index.html
++ https://scala3doc.s3.eu-central-1.amazonaws.com/pr-123/stdLib/main/index.html
+
+Note that these correspond to the contents of `output` directory - that's
+precisely what they are.
+
+You can also find the result of building the same sites for latest `master` at:
+
++ https://scala3doc.s3.eu-central-1.amazonaws.com/pr-master/self/main/index.html
++ https://scala3doc.s3.eu-central-1.amazonaws.com/pr-master/stdLib/main/index.html
+
+### Testing
+
+Most tests rely on comparing signatures (of classes, methods, objects etc.) extracted from the generated documentation
+to signatures found in source files. Such tests are defined using [MultipleFileTest](src/test/scala/dotty/dokka/MultipleFileTest.scala) class
+and its subtypes (such as [SingleFileTest](src/test/scala/dotty/dokka/SingleFileTest.scala))
+
+WARNING: As the classes mentioned above are likely to evolve, the description below might easily get out of date.
+In case of any discrepancies rely on the source files instead.
+
+`MultipleFileTest` requires that you specify the names of the files used to extract signatures,
+the names of directories containing corresponding TASTY files
+and the kinds of signatures from source files (corresponding to keywords used to declare them like `def`, `class`, `object` etc.)
+whose presence in the generated documentation will be checked (other signatures, when missing, will be ignored).
+The mentioned source files should be located directly inside `src/main/scala/tests` directory
+but the file names passed as parameters should contain neither this path prefix nor `.scala` suffix.
+The TASTY folders are expected to be located in `target/${dottyVersion}/classes/tests` (after successful compilation of the project)
+and similarly only their names relative to this path should be provided as tests' parameters.
+For `SingleFileTest` the name of the source file and the TASTY folder are expected to be the same.
+
+By default it's expected that all signatures from the source files will be present in the documentation
+but not vice versa (because the documentation can contain also inherited signatures).
+To validate that a signature present in the source does not exist in the documentation
+(because they should be hidden from users) add `//unexpected` comment after the signature in the same line.
+This will cause an error if a signature with the same name appears in the documentation
+(even if some elements of the signature are slightly different - to avoid accidentally passing tests).
+If the signature in the documentation is expected to slightly differ from how it's defined in the source code
+you can add a `//expected: ` comment (also in the same line and followed by a space) followed by the expected signature.
+Alternatively you can use `/*<-*/` and `/*->*/` as opening and closing parentheses for parts of a signature present in the source but undesired in the documentation (at least at the current stage of development), e.g.
+
+```
+def foo/*<-*/()/*->*/: Int
+```
+
+will make the expected signature be
+
+```
+def foo: Int
+```
+
+instead of
+
+```
+def foo(): Int
+```
+
+
+Because of the way how signatures in source are parsed, they're expected to span until the end of a line (including comments except those special ones mentioned above, which change the behaviour of tests) so if a definition contains an implementation, it should be placed in a separate line, e.g.
+
+```
+def foo: Int
+   = 1
+
+class Bar
+{
+   //...
+}
+```
+
+Otherwise the implementation would be treated as a part of the signature.
+
+## Roadmap
+
+1. Publish an initial version of the tool together with an SBT plugin
+1. Replace Dottydoc as the dedicated tool for documenting Dotty code
+
+   This includes:
+   + supporting Dotty's doc pages
+   + releasing together with Dotty as the dedicated documentation tool
+
+1. Support all kinds of Dotty definition and generate documentation for the
+   standard library
+1. Reach feature parity with Scaladoc
+
+## Contributing
+
+We're happy that you'd like to help us!
+
+We have two issue labels you should take a look at: `good first issue` and
+`self-contained`. First is easy pickings: you'll be able to contribute without
+needing to dive too deep into the project. Second is reverse: it's an issue
+that's you may find interesting, complex and self-contained enough that you can
+continue chipping away at it without needing to worry too much about merge
+conflicts.
+
+To contribute to the project with your code, fork this repo and create a pull request from a fresh branch from there.
+To keep the history of commits clean, make sure your commits are squashed into one
+and all your changes are applied on top of the latest master branch (if not - rebase on it instead of merging it).
+Make sure all the tests pass (simply run `sbt test` to verify that).
+
+## FAQ
+
+### Why depend on Dokka?
+
+We have two primary reasons for depending on Dokka. One of them is division of
+labour - Dokka already has a team of maintainers, and it supports an excellent
+API which already allowed us to quite easily generate documentation with it. By
+depending on Dokka, we will be able to share a large portion of the maintenance
+burden. The second reason is very pragmatic - on our own, it'd be difficult for
+us to reach even feature parity with Scaladoc, simply because of workforce
+constraints. Meanwhile, Dokka maintainers from VirtusLab reached out to us with
+an offer of help, which we were happy to take.
+
+### Why use TASTy?
+
+A documentation tool needs to access compiler information about the project - it
+needs to list all definitions, resolve them by name, and query their members.
+Tasty Reflect is the dedicated way in Scala 3 of accessing this information.

scala3doc/build.sbt

@@ -0,0 +1,75 @@
+// re-expose subproject settings
+val `example-project` = ExampleProject.`example-project`
+
+val dottyVersion = "0.27.0-RC1"
+val dokkaVersion = "1.4.10.2"
+val flexmarkVersion = "0.42.12"
+val jacksonVersion = "2.9.8"
+val scalaTagsVersion = "0.9.1"
+val dokkaSiteVersion = "0.1.9"
+
+libraryDependencies ++= Seq(
+  "org.jetbrains.dokka" % "dokka-test-api" % dokkaVersion % "test", // TODO move testing utils to dokka-site
+  "com.virtuslab.dokka" % "dokka-site" % dokkaSiteVersion,
+
+  "ch.epfl.lamp" %% "dotty-tasty-inspector" % dottyVersion,
+  "ch.epfl.lamp" %% "dotty-compiler" % dottyVersion,
+  "ch.epfl.lamp" %% "dotty-library" % dottyVersion,
+  "org.scala-sbt" % "io_2.13" % "1.3.4",
+
+  "com.vladsch.flexmark" % "flexmark-all" % flexmarkVersion,
+  "com.lihaoyi" % "scalatags_2.13" % scalaTagsVersion,
+  "nl.big-o" % "liqp" % "0.6.7",
+  "args4j" % "args4j" % "2.33",
+  "com.novocode" % "junit-interface" % "0.11" % "test",
+)
+
+resolvers += Resolver.jcenterRepo
+resolvers += Resolver.bintrayRepo("kotlin", "kotlin-dev")
+resolvers += Resolver.bintrayRepo("virtuslab", "dokka")
+resolvers += Resolver.mavenLocal
+
+lazy val root = project
+  .in(file("."))
+  .settings(
+    name := "scala3doc",
+    version := "0.1.1-SNAPSHOT",
+    scalaVersion := dottyVersion
+  )
+
+val generateSelfDocumentation = inputKey[Unit]("Generate example documentation")
+generateSelfDocumentation := {
+  run.in(Compile).fullInput(" -o output/self -t target/scala-0.27/classes -d documentation -n scala3doc -s src/main/scala=https://github.com/lampepfl/scala3doc/tree/master/src/main/scala#L").evaluated // TODO #35 proper sbt integration
+}
+
+// Uncomment to debug dokka processing (require to run debug in listen mode on 5005 port)
+// javaOptions.in(run) += "-agentlib:jdwp=transport=dt_socket,server=n,address=localhost:5005,suspend=y"
+
+fork.in(run) := true
+// There is a bug in dokka that prevents parallel tests withing the same jvm
+fork.in(test) := true
+Test / parallelExecution := false
+
+scalacOptions in Compile += "-language:implicitConversions"
+
+Compile / mainClass := Some("dotty.dokka.Main")
+
+// hack, we cannot build documentation so we need this to publish locally
+publishArtifact in (Compile, packageDoc) := false
+
+// TODO #35 proper sbt integration
+val generateDottyLibDocumentation = taskKey[Unit]("Generate documentation for dotty lib")
+generateDottyLibDocumentation :=  Def.taskDyn {
+  val dotttyLib = fullClasspath.in(Compile).value.find{ a =>
+    val info = a.get(moduleID.key)
+    info.nonEmpty &&
+     info.get.organization == "ch.epfl.lamp" &&
+     info.get.name.startsWith("dotty-library")
+  }
+  if (dotttyLib.isEmpty) Def.task {
+    streams.value.log.error("Dotty lib wasn't found")
+  } else Def.task {
+    run.in(Compile).toTask(s" -o output/stdLib -t ${dotttyLib.get.data} -d dotty-docs/docs -n dotty-lib -s library/src=https://github.com/lampepfl/dotty/tree/master/library/src#L").value
+  } 
+}.value
+

scala3doc/documentation/docs/index.md

@@ -0,0 +1 @@
+../../README.md

scala3doc/documentation/index.md

@@ -0,0 +1,27 @@
+# Scala3doc
+
+**Documentation tool for Scala 3**
+
+We are using [TASTy](https://github.com/lampepfl/dotty/blob/master/tasty/src/dotty/tools/tasty/TastyFormat.scala) to generate documentation. We aim to has all known and loved feature from scaladoc as well as new feature such as :
+
+- integrated documentation and API
+- has option for basic pluggablity
+- and much more
+
+**Yes, this page was generated using scala3doc**
+
+You can learn more from out [documentation](scala3doc/index.html).
+
+## Getting started
+
+For now the recommended way to try out our project would be:
+ - Clone our [repository](https://github.com/lampepfl/scala3doc)
+ - Run `sbt main -n <name> -o <output> -t <tasty-files> -cp <classpath> -s { <sources> } ` where
+   - `<name>`: name of module in generated documentation
+   - `<output>`: location where documentation should be created
+   - `<tasty-files>`: is list of dirs or jars that contains tasty files that should be documented
+   - `<classpath>`: classpath that was used to generate tasty files
+   - `<sources>`: links to source files of module that are used to link symbols on pages to their source file. They need to be supplied in form:
+      local_dir=remote_dir#line_suffix e.g. src/main/scala=https://github.com/lampepfl/scala3doc/tree/master/src/main/scala#L
+
+   We also support `-d <documentation>` argument to provide static documentation. You can find more about that feature [here](static-page.html).

scala3doc/dotty-docs/docs/.gitignore

@@ -0,0 +1,6 @@
+# Jekyll specific ignores
+vendor/
+.bundle/
+Gemfile.lock
+_site/
+.sass-cache/