Replace apache download tasks with custom ivy repository. #1438
Conversation
Hadoop ecosystem tarballs are hosted on Apache's download mirrors only instead of in maven repositories. Previously we were downloading these artifacts from a round robin of mirror endpoints, using securely obtained hashes to validate the downloaded files even from non HTTPS sources. This PR removes those download tasks, replacing it with a custom Ivy repository backed by an HTTPS enabled apache mirror. This allows us to ensure the artifacts are securely downloaded and cached by Gradle instead of relying on task outputs like the old mirror download task.
|
One thing to keep in mind here is that Gradle doesn't do artifact checksum verification by default. In fact, this capability was only recently added in Gradle 6.2. As it is, this PR removes the checksum validation for these artifacts. If we want to retain that capability, we'll need to enabled checksum verification in Gradle as described here. |
|
The dependency verification features look really useful, but I think they might be a bit out of scope. It think we mostly want the verification for downloads from apache mirrors, but it's a global setting that will require upkeep outside of normal project configuration. Additionally, we would need to maintain a number of unverified configurations since we rely on the local file repo in snapshot and release builds. |
...rc/src/main/groovy/org/elasticsearch/hadoop/gradle/fixture/hadoop/HadoopFixturePlugin.groovy
Outdated
Show resolved
Hide resolved
Hadoop ecosystem tarballs are hosted on Apache's download mirrors only instead of in maven repositories. Previously we were downloading these artifacts from a round robin of mirror endpoints, using securely obtained hashes to validate the downloaded files even from non HTTPS sources. This PR removes those download tasks, replacing it with a custom Ivy repository backed by an HTTPS enabled apache mirror. This allows us to ensure the artifacts are securely downloaded and cached by Gradle instead of relying on task outputs like the old mirror download task.
Hadoop ecosystem tarballs are hosted on Apache's download mirrors only instead of in maven repositories. Previously we were downloading these artifacts from a round robin of mirror endpoints, using securely obtained hashes to validate the downloaded files even from non HTTPS sources.
This PR removes those download tasks, replacing it with a custom Ivy repository backed by an HTTPS enabled apache mirror. This allows us to ensure the artifacts are securely downloaded and cached by Gradle instead of relying on task outputs like the old mirror download task.