Gradle builder for Java #69

dagnir · 2019-01-09T23:28:10Z

Issue #, if available:

Description of changes:
This adds the design document to enable sam build for Gradle based Java projects.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

aws_lambda_builders/workflows/java_gradle/DESIGN.md

abrooksv · 2019-01-15T18:55:57Z

aws_lambda_builders/workflows/java_gradle/DESIGN.md

-compiled classes, resouces, and dependencies into a ZIP.
+### Source directory, and artifact directory semantics
+
+To enable the multi build projects where a Gradle project can contain multiple


I know we did not want to discuss this in relation to sam, but I feel we need to in this case to understand what the UX is for users of sam build.

@jfuss Will this be exposed through sam build that users will need to understand and set? Or will sam build hide this?

This will be hidden. Using SAM CLI as an example:

A customer needing this could:
sam build -m ./build.gradle

SAM CLI will then conform to the details here. So we will provide the correct source_dir and artifact_dir. For the use-container, we will need to mount all of ProjectB (only way this will work, I think).

SAM CLI will only do this operation if -m is provided and the runtime is Java. Otherwise, SAM CLI will assume the Project A is what is desired (aka building the individual functions only).

abrooksv · 2019-01-15T19:04:21Z

aws_lambda_builders/workflows/java_gradle/DESIGN.md

+If this map is present, then `source_dir` is treated as the root directory of
+the parent project. Additionally, `artifact_dir` will not be the path under
+which the artifact will be copied to, but just the parent of the final
+directory. `artifact_mapping` will be a mapping from a source subdirectory under


If the artifact_mapping is provided, should we optimize the build by doing

for each mapping in artifact_mapping: cmd =+ " $mapping.folderName:build" $GRADLE_EXECUTABLE --init-script /$SCRATCH_DIR/lambda-build-init.gradle $cmd

This does lead to issues where people make the sub-project name not match the folder (why people would I do not know...)
In that case we may have to make init script manipulare the task graph based on the project root of each sub-project. It's possible, just question around "should we"

We can always optimize later if this is really needed.

aws_lambda_builders/workflows/java_gradle/DESIGN.md

- Add note about Project A being the function

- Remove unneeded wait() - Disable daemon, seems to cause a hang when shelling out to Gradle

aws_lambda_builders/workflows/java_gradle/gradle.py

aws_lambda_builders/workflows/java_gradle/actions.py

aws_lambda_builders/workflows/java_gradle/workflow.py

- Use resolvers for finding Gradle executable - Extract copy artifacts into separate action - Add tests for mapping

Gradle can be configured to explicitly run with a specific JVM which may not be the one found on the PATH. Getting the JVM version from Gradle (i.e. the JVM version that Gradle is running in) helps us be more accurate with the warning.

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle

...ration/workflows/java_gradle/testdata/multi-build/with-deps-inter-module/common/build.gradle

- Use archives artifacts to find project class files - Add extra tests for projects with resources and test dependencies

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle

tests/integration/workflows/java_gradle/test_java_gradle.py

aws_lambda_builders/workflows/java_gradle/DESIGN.md

- Builder only needs to be concerned with building `source_dir` so remove `artifact_mapping` semantics - Ensure buildDir is on scratch_dir - Ensure project-cache-dir in scratch_dir

sriram-mv · 2019-02-11T17:21:56Z

There is a PR out for searching additional paths when looking for executables that are to be used by the workflow. this will help for locating gradlew at different paths. #77 . #69 needs to integrate that.

dagnir · 2019-02-11T17:37:34Z

@thesriram Looks like it's #78? Is it mostly ready to go? Or should I wait for it to be merged?

sanathkr

This is a really cool work! Thanks for the detailed design document. Makes it really easy to understand the code. Most of my comments are general cleanup, stylistic and questions. Feel free to push back if it doesn't make sense.

.appveyor.yml

aws_lambda_builders/workflows/java_gradle/DESIGN.md

aws_lambda_builders/workflows/java_gradle/workflow.py

aws_lambda_builders/workflows/java_gradle/gradle_resolver.py

aws_lambda_builders/workflows/java_gradle/gradle_validator.py

sanathkr · 2019-02-11T22:28:43Z

tests/integration/workflows/java_gradle/test_java_gradle.py

+from aws_lambda_builders.exceptions import WorkflowFailedError
+
+
+class TestJavaGradle(TestCase):


Love the extensive coverage of integration tests! Thanks a lot for taking the time. Really helps pushing the boundary for code quality

When the executable has an error return code when we run 'gradle -v', we also treat that as a "version not found".

sanathkr

The general logic makes sense, but I want to find a way to achieve same result without the dummy values and exception message parsing. If needed, we can change PathResolver design to make this implementaiton simper

aws_lambda_builders/workflows/java_gradle/DESIGN.md

aws_lambda_builders/workflows/java_gradle/gradlew_resolver.py

aws_lambda_builders/workflows/java_gradle/gradle_validator.py

aws_lambda_builders/workflows/java_gradle/workflow.py

aws_lambda_builders/workflows/java_gradle/gradlew_resolver.py

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle

aws_lambda_builders/workflows/java_gradle/workflow.py

- Fix gradlew path resolution to not rely on PathResolver - Write class files and libraries directly to artifact_dir root instead of creating a ZIP - Raise exceptin if manifest file not found

sanathkr

Appreciate cleaning this up. There is one thing that won't work in Py2. Can you fix it?

sanathkr · 2019-02-14T15:11:34Z

aws_lambda_builders/workflows/java_gradle/actions.py

@@ -72,7 +76,13 @@ def execute(self):
    def _copy_artifacts(self):
        lambda_build_output = os.path.join(self.build_dir, 'build', 'distributions', 'lambda-build')
        try:
-            for f in self.os_utils.listdir(lambda_build_output):
-                self.os_utils.copy(os.path.join(lambda_build_output, f), self.artifacts_dir)
+            self.os_utils.makedirs(self.artifacts_dir)


os.scandir is not available in Python 2.7. Can you use this osutils helper instead?

https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/utils.py#L56-L66

Copy/paste it in your code for now.

sanathkr

You made it! 💥Thanks a lot for diving deep to get the design right and all the revisions to improve implementation.

sriram-mv

💃

sriram-mv · 2019-02-14T19:11:00Z

tests/integration/workflows/java_gradle/test_java_gradle.py

+        self.assert_artifact_contains_files(expected_files)
+        self.assert_artifact_not_contains_file(p('lib', 's3-2.1.0.jar'))
+
+    def test_build_single_build_with_deps_gradlew(self):


Nice! executable search paths :)

2 approvals from sriram & sanath

Add design for Gradle builder for Java

1628f95

abrooksv reviewed Jan 10, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Outdated Show resolved Hide resolved

aws_lambda_builders/workflows/java_gradle/DESIGN.md Outdated Show resolved Hide resolved

abrooksv reviewed Jan 10, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Outdated Show resolved Hide resolved

sriram-mv reviewed Jan 10, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Show resolved Hide resolved

abrooksv reviewed Jan 10, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Show resolved Hide resolved

jfuss suggested changes Jan 11, 2019

View reviewed changes

Address review comments

ac6e20a

abrooksv reviewed Jan 15, 2019

View reviewed changes

jfuss reviewed Jan 15, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Show resolved Hide resolved

dagnir added 7 commits January 18, 2019 15:15

Review comments

b80b2c8

- Add note about Project A being the function

Implement Java Gradle Builder

2209483

Merge remote-tracking branch 'origin/develop' into java_gradle_design

7740a87

Fix formatting

264fef6

Linter fixes

badaacc

Install Gradle on AppVeyor

e099521

AppVeyor build fixes

581caa8

- Remove unneeded wait() - Disable daemon, seems to cause a hang when shelling out to Gradle

jfuss previously requested changes Jan 29, 2019

View reviewed changes

dagnir added 4 commits January 29, 2019 17:52

Review comments

e9a856f

- Use resolvers for finding Gradle executable - Extract copy artifacts into separate action - Add tests for mapping

Add Java version validation

b1ec1ea

Fix version regex

a5a41ee

Move JVM version validation to gradle validation

3db3709

Gradle can be configured to explicitly run with a specific JVM which may not be the one found on the PATH. Getting the JVM version from Gradle (i.e. the JVM version that Gradle is running in) helps us be more accurate with the warning.

abrooksv reviewed Jan 31, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle Outdated Show resolved Hide resolved

...ration/workflows/java_gradle/testdata/multi-build/with-deps-inter-module/common/build.gradle Show resolved Hide resolved

Review comments

f4a855d

- Use archives artifacts to find project class files - Add extra tests for projects with resources and test dependencies

abrooksv reviewed Feb 1, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle Outdated Show resolved Hide resolved

abrooksv reviewed Feb 1, 2019

View reviewed changes

tests/integration/workflows/java_gradle/test_java_gradle.py Show resolved Hide resolved

Cleanup init file

27ba9a2

sanathkr reviewed Feb 8, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/DESIGN.md Outdated Show resolved Hide resolved

aws_lambda_builders/workflows/java_gradle/DESIGN.md Outdated Show resolved Hide resolved

Design/Impl updates per feedback

6d4944f

- Builder only needs to be concerned with building `source_dir` so remove `artifact_mapping` semantics - Ensure buildDir is on scratch_dir - Ensure project-cache-dir in scratch_dir

sanathkr suggested changes Feb 11, 2019

View reviewed changes

dagnir added 4 commits February 12, 2019 11:36

Clarify project subdirectory usage under scratch_dir

e575c96

Emit warning when version string not found

82c9db6

When the executable has an error return code when we run 'gradle -v', we also treat that as a "version not found".

Merge remote-tracking branch 'origin/develop' into java_gradle_design

00d71c6

Use aws#78 for Gradle Wrapper resolution

1ba3c4a

sanathkr suggested changes Feb 13, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/resources/lambda-build-init.gradle Outdated Show resolved Hide resolved

sanathkr suggested changes Feb 13, 2019

View reviewed changes

aws_lambda_builders/workflows/java_gradle/workflow.py Show resolved Hide resolved

Review comments

ef8c959

- Fix gradlew path resolution to not rely on PathResolver - Write class files and libraries directly to artifact_dir root instead of creating a ZIP - Raise exceptin if manifest file not found

sanathkr suggested changes Feb 14, 2019

View reviewed changes

sanathkr changed the title ~~Add design for Gradle builder for Java~~ Gradle builder for Java Feb 14, 2019

sanathkr mentioned this pull request Feb 14, 2019

Add support for .NET Core build #80

Closed

dagnir added 2 commits February 14, 2019 10:25

Remove scandir; not avaialble in Python 2

b74ea79

More Python 2 fixes

37a397e

sanathkr approved these changes Feb 14, 2019

View reviewed changes

sriram-mv approved these changes Feb 14, 2019

View reviewed changes

sanathkr merged commit 8415ccd into aws:develop Feb 14, 2019

		from aws_lambda_builders.exceptions import WorkflowFailedError


		class TestJavaGradle(TestCase):

Gradle builder for Java #69

Gradle builder for Java #69

Uh oh!

Conversation

dagnir commented Jan 9, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

abrooksv Jan 15, 2019

Choose a reason for hiding this comment

Uh oh!

jfuss Jan 15, 2019

Choose a reason for hiding this comment

Uh oh!

abrooksv Jan 15, 2019

Choose a reason for hiding this comment

Uh oh!

jfuss Jan 15, 2019

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sriram-mv commented Feb 11, 2019

Uh oh!

dagnir commented Feb 11, 2019

Uh oh!

sanathkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sanathkr Feb 11, 2019

Choose a reason for hiding this comment

Uh oh!

sanathkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sanathkr left a comment

Choose a reason for hiding this comment

Uh oh!

sanathkr Feb 14, 2019

Choose a reason for hiding this comment

Uh oh!