Use different Ubuntu pools for Ubuntu tests to avoid disk space issues. #20742

NTaylorMullen · 2020-04-10T23:21:06Z

Updated the default-build.yml to have a new parameter specific to Ubuntu listed useHostedUbuntu. If we feel the need to expand the parameters usage outside of the Ubuntu configuration we can always rename to useHosted. Also didn't want to touch the isTestingJob because I wasn't sure of hte implications.
Updated the Ubuntu test job to turn off hosted pools.

Fixes https://github.com/dotnet/aspnetcore-internal/issues/3574

dougbu

Much appreciated! Let's hope we don't find these agents are missing much.

Eventually, it'll be interesting to get #20704 in and see how much space these guys have lying about.

- Updated the `default-build.yml` to have a new parameter specific to Ubuntu listed `useHostedUbuntu`. If we feel the need to expand the parameters usage outside of the Ubuntu configuration we can always rename to `useHosted`. Also didn't want to touch the `isTestingJob` because I wasn't sure of hte implications. - Updated the Ubuntu test job to turn off hosted pools. Fixes https://github.com/dotnet/aspnetcore-internal/issues/3574

.azure/pipelines/jobs/default-build.yml

NTaylorMullen · 2020-04-10T23:38:49Z

@dougbu are those pools/queues correct? It looks like it's attempting to run our Linux scripts on a windows box for the Linux jobs:

dougbu · 2020-04-10T23:42:19Z

.azure/pipelines/jobs/default-build.yml

+            queue: buildpool.ubuntu.1604.amd64.open
+          ${{ if eq(variables['System.TeamProject'], 'internal') }}:
+            name: NetCoreInternal-Pool
+            queue: buildpool.ubuntu.1604.amd64


@ilyas1974 and @riarenas how are we messing up here? These jobs are trying to run on Windows agents.

Is useHostedUbuntu somehow undefined here so it is using the default?

My suspicion based on https://helix.dot.net/api/2019-06-17/info/queues (now that I look closely) is queue names are case-sensitive. I just pushed a commit to see if that's the case.

.azure/pipelines/ci.yml

Pilchie · 2020-04-11T00:07:00Z

.azure/pipelines/jobs/default-build.yml

+            queue: buildpool.ubuntu.1604.amd64.open
+          ${{ if eq(variables['System.TeamProject'], 'internal') }}:
+            name: NetCoreInternal-Pool
+            queue: buildpool.ubuntu.1604.amd64


Is useHostedUbuntu somehow undefined here so it is using the default?

dougbu · 2020-04-11T00:16:22Z

.azure/pipelines/jobs/default-build.yml

@@ -83,6 +83,7 @@ jobs:
    enableTelemetry: true
    helixRepo: dotnet/aspnetcore
    helixType: build.product/
+    useHostedUbuntu: true


Whether or not queue names are case-sensitive, this placement was the root cause. I didn't remember that this section is about passing parameters into the job.yml template when talking to @NTaylorMullen about this. This parameter default needs to be near the top of the file ☹️

dougbu · 2020-04-11T00:20:00Z

Is useHostedUbuntu somehow undefined here so it is using the default?

Yes, that was broken too as I mentioned here. My bad.

Pilchie · 2020-04-11T00:58:43Z

Strange that some server tests are failing here, with a file not found.

Also this warning in the build:
https://dev.azure.com/dnceng/public/_build/results?buildId=597050&view=logs&j=b72e85ab-3386-5aa9-6405-3837662d9688&t=f7506d25-98a6-55c7-2099-fa268bf83c45&l=14

warning: Some managed projects depend on NodeJS projects. Building NodeJS is disabled so the managed projects will fallback to using the output from previous builds. The output may not be correct or up to date.

dougbu · 2020-04-11T01:29:09Z

@Pilchie thanks for digging into this!

@ilyas1974 and @riarenas, I'll see if the missing file errors are all related to not having node installed and let you know if anything else is missing from the images that we need. If it's just that tool, I'll add the node install task in the problem child job.

dougbu · 2020-04-11T03:01:29Z

I'm not sure why the build said it didn't find node but suspect it's not the core problem. Instead, the nginx installation failed because it couldn't find PCRE:

checking for PCRE library ... not found
checking for PCRE library in /usr/local/ ... not found
checking for PCRE library in /usr/include/pcre/ ... not found
checking for PCRE library in /usr/pkg/ ... not found
checking for PCRE library in /opt/local/ ... not found

./configure: error: the HTTP rewrite module requires the PCRE library.
You can either disable the module by using --without-http_rewrite_module
option, or install the PCRE library into the system, or build the PCRE library
statically from the source with nginx by using --with-pcre=<path> option.

make: *** No rule to make target 'build', needed by 'default'.  Stop.
make: *** No rule to make target 'install'.  Stop.

@BrennanConroy @anurse do we actually need the HTTP rewrite model? Otherwise, what package should we ask for? https://packages.ubuntu.com/xenial/pcre2-utils looks close but more low level than the agents usually deal with.

Of course, continuing after the error implies eng/scripts/install-nginx-linux.sh and probably install-nginx-mac.sh need set -euo pipefail. (Does this command work on macOS?)

BrennanConroy · 2020-04-11T03:17:53Z

I doubt we need http rewrite

- also add `set -euo pipefail` to the script for fast failure

dougbu · 2020-04-11T05:19:06Z

Wow, I think we're done with the Ubuntu test job timing out 🚀 Thanks @NTaylorMullen @ilyas1974 @Pilchie @BrennanConroy @riarenas @pranavkm @MattGal @markwilkie and anyone I'm forgetting for your help❕

I'll get this in tonight or tomorrow morning, depending on the remaining two (unchanged) build jobs.

NTaylorMullen requested review from pranavkm, Pilchie and dougbu April 10, 2020 23:21

ghost added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Apr 10, 2020

dougbu approved these changes Apr 10, 2020

View reviewed changes

NTaylorMullen force-pushed the nimullen/changeubuntupools branch from fc86c36 to 7e7c9b4 Compare April 10, 2020 23:25

dougbu reviewed Apr 10, 2020

View reviewed changes

.azure/pipelines/jobs/default-build.yml Show resolved Hide resolved

dougbu reviewed Apr 10, 2020

View reviewed changes

dougbu added 2 commits April 10, 2020 17:01

Hmm, are queue names case-sensitive?

be2d579

Correct useHostedUbuntu: false placement

d678a5a

Pilchie reviewed Apr 11, 2020

View reviewed changes

dougbu reviewed Apr 11, 2020

View reviewed changes

Correct placement of useHostedUbuntu: true default

989a6de

Try adding --without-http_rewrite_module to nginx configuration

45185d3

- also add `set -euo pipefail` to the script for fast failure

dougbu merged commit 51f69a6 into master Apr 11, 2020

dougbu deleted the nimullen/changeubuntupools branch April 11, 2020 06:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use different Ubuntu pools for Ubuntu tests to avoid disk space issues. #20742

Use different Ubuntu pools for Ubuntu tests to avoid disk space issues. #20742

Uh oh!

NTaylorMullen commented Apr 10, 2020

Uh oh!

dougbu left a comment

Uh oh!

Uh oh!

NTaylorMullen commented Apr 10, 2020

Uh oh!

dougbu Apr 10, 2020

Uh oh!

Pilchie Apr 11, 2020

Uh oh!

dougbu Apr 11, 2020

Uh oh!

Uh oh!

Pilchie Apr 11, 2020

Uh oh!

dougbu Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

Pilchie commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

BrennanConroy commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

Uh oh!

Use different Ubuntu pools for Ubuntu tests to avoid disk space issues. #20742

Use different Ubuntu pools for Ubuntu tests to avoid disk space issues. #20742

Uh oh!

Conversation

NTaylorMullen commented Apr 10, 2020

Uh oh!

dougbu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NTaylorMullen commented Apr 10, 2020

Uh oh!

dougbu Apr 10, 2020

Choose a reason for hiding this comment

Uh oh!

Pilchie Apr 11, 2020

Choose a reason for hiding this comment

Uh oh!

dougbu Apr 11, 2020

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Pilchie Apr 11, 2020

Choose a reason for hiding this comment

Uh oh!

dougbu Apr 11, 2020

Choose a reason for hiding this comment

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

Pilchie commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

BrennanConroy commented Apr 11, 2020

Uh oh!

dougbu commented Apr 11, 2020

Uh oh!

Uh oh!