Skip to content

Commit 5377efa

Browse files
adriaanmadriaanm
committed
wip: artifactory and required cleanups
1 parent 6723844 commit 5377efa

11 files changed

+377
-266
lines changed

‎.chef/Vagrantfile

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,20 @@
11
# TODO: this kind of works, but the cookbook won't run in virtual box:
2-
# - (scatter-gather workaround)
32
# - chef-solo can't access the vault on the chef server (passwords etc)
4-
# to make this work, you need to populate a cookbooks/ directory next to this file,
5-
# as documented in the README.md (use knife)
3+
# vagrant box add utopic-daily https://cloud-images.ubuntu.com/vagrant/utopic/current/utopic-server-cloudimg-amd64-vagrant-disk1.box
4+
# vagrant init utopic-daily
5+
# vagrant up
6+
# centos: vagrant box add chef/centos-7.0
67
Vagrant.configure("2") do |config|
7-
config.vm.box = "hashicorp/precise64"
8+
config.vm.box = "utopic-daily"
89
config.vm.provision :chef_solo do |chef|
10+
chef.cookbooks_path = ["~/git/cookbooks"]
11+
chef.node_name = "jenkins-master"
912
chef.add_recipe("scala-jenkins-infra::master-init")
13+
chef.add_recipe("scala-jenkins-infra::_master-config-proxy")
14+
end
15+
config.vm.network "public_network"
16+
config.vm.provider "virtualbox" do |v|
17+
v.memory = 4096
18+
v.cpus = 2
1019
end
11-
config.vm.network :forwarded_port, guest: 80, host: 11180
1220
end

‎README.md

Lines changed: 54 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -169,30 +169,64 @@ Test if knife works correctly by running `knife cookbook list`.
169169

170170
Obtain the organization validation key from Adriaan and put it to `.chef/config/$CHEF_ORG-validator.pem`. (Q: When is this key used exactly? https://docs.chef.io/chef_private_keys.html says it's when a new node runs `chef-client` for the first time.)
171171

172-
## Get cookbooks
172+
## Clone scala-jenkins-infra cookbook and its dependencies
173+
174+
I think you can safely ignore `ERROR: IOError: Cannot open or read **/metadata.rb!`
173175

174176
```
175-
git init .chef/cookbooks
176-
cd .chef/cookbooks
177+
cd ~/git/cookbooks
178+
git init .
177179
g commit --allow-empty -m"Initial"
178-
```
179-
180-
- knife cookbook site install wix 1.0.2 # newer versions don't work for me; also installs windows
181-
- knife cookbook site install aws
182-
- knife cookbook site install git
183-
- knife cookbook site install git_user
184-
- knife cookbook site install partial_search
185-
- knife cookbook site install artifactory
186180
187-
- move to unreleased versions on github:
188-
- knife cookbook github install opscode-cookbooks/windows # fix nosuchmethoderror (#150)
189-
- knife cookbook github install adriaanm/java/windows-jdk1.6 # jdk 1.6 installer barfs on re-install -- wipe its INSTALLDIR
190-
- knife cookbook github install adriaanm/jenkins/fix305 # ssl fail on windows
191-
- knife cookbook github install adriaanm/scala-jenkins-infra
192-
- knife cookbook github install adriaanm/chef-sbt
193-
- knife cookbook github install gildegoma/chef-sbt-extras
194-
195-
- knife cookbook upload --all
181+
hub clone scala/scala-jenkins-infra
182+
cd scala-jenkins-infra
183+
ln -sh ~/git/cookbooks .chef/
184+
185+
knife site install cron
186+
knife site install logrotate
187+
knife site install chef_handler
188+
knife site install windows
189+
knife site install chef-client
190+
knife site install aws
191+
knife site install delayed_evaluator
192+
knife site install ebs
193+
knife site install java
194+
knife site install apt
195+
knife site install packagecloud
196+
knife site install runit
197+
knife site install yum
198+
knife site install jenkins
199+
knife site install 7-zip
200+
knife site install ark
201+
knife site install artifactory
202+
knife site install build-essential
203+
knife site install dmg
204+
knife site install yum-epel
205+
knife site install git
206+
knife site install user
207+
knife site install partial_search
208+
knife site install ssh_known_hosts
209+
knife site install git_user
210+
knife site install chef-sbt
211+
knife site install sbt-extras
212+
```
213+
214+
### Switch to unreleased versions from github
215+
```
216+
//fixed: knife cookbook github install opscode-cookbooks/windows # fix nosuchmethoderror (#150)
217+
//knife cookbook github install adriaanm/jenkins/fix305 # ssl fail on windows -- fix pending: https://github.com/opscode-cookbooks/jenkins/pull/313
218+
knife cookbook github install b-dean/jenkins/http_ca_fixes # pending fix for above ^^^
219+
220+
knife cookbook github install adriaanm/java/windows-jdk1.6 # jdk 1.6 installer barfs on re-install -- wipe its INSTALLDIR
221+
knife cookbook github install adriaanm/chef-sbt
222+
knife cookbook github install gildegoma/chef-sbt-extras
223+
knife cookbook github install adriaanm/artifactory
224+
```
225+
226+
### Upload cookbooks to chef server
227+
```
228+
knife cookbook upload --all
229+
```
196230

197231
## Cache installers locally
198232
- they are tricky to access, might disappear,...

‎attributes/master.rb

Lines changed: 88 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -1,77 +1,99 @@
1-
# EBS
2-
default['ebs']['volumes']['/var/lib/jenkins']['size'] = 100 # size of the volume correlates to speed (in IOPS)
3-
default['ebs']['volumes']['/var/lib/jenkins']['dev'] = "/dev/sdj"
4-
default['ebs']['volumes']['/var/lib/jenkins']['fstype'] = "ext4"
5-
default['ebs']['volumes']['/var/lib/jenkins']['user'] = "jenkins"
6-
default['ebs']['volumes']['/var/lib/jenkins']['mountopts'] = 'noatime'
1+
scalaCiHost = "scala-ci.typesafe.com"
2+
scalaCiPort = 443
73

8-
default['ebs']['volumes']['/var/lib/artifactory']['size'] = 200 # size of the volume correlates to speed (in IOPS)
9-
default['ebs']['volumes']['/var/lib/artifactory']['dev'] = "/dev/sdk"
10-
default['ebs']['volumes']['/var/lib/artifactory']['fstype'] = "ext4"
11-
default['ebs']['volumes']['/var/lib/artifactory']['user'] = "artifactory"
12-
default['ebs']['volumes']['/var/lib/artifactory']['mountopts'] = 'noatime'
4+
# JENKINS WORKER CONFIG
5+
default['repos']['private']['realm'] = "Artifactory Realm"
6+
default['repos']['private']['host'] = "private-repo.typesafe.com"
7+
default['repos']['private']['pr-snap'] = "http://private-repo.typesafe.com/typesafe/scala-pr-validation-snapshots/"
8+
default['repos']['private']['release-temp'] = "http://private-repo.typesafe.com/typesafe/scala-release-temp/"
9+
default['s3']['downloads']['host'] = "downloads.typesafe.com.s3.amazonaws.com"
1310

14-
# JENKINS
15-
override['jenkins']['master']['install_method'] = 'war'
16-
override['jenkins']['master']['listen_address'] = '127.0.0.1' # external traffic must go through nginx
17-
override['jenkins']['master']['user'] = 'jenkins'
18-
override['jenkins']['master']['group'] = 'jenkins'
19-
override['jenkins']['master']['jvm_options'] = '-server -Xmx4G -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError' # -Dfile.encoding=UTF-8
11+
if node.name == "jenkins-master"
12+
# EBS
13+
default['ebs']['volumes']['/var/lib/jenkins']['size'] = 100 # size of the volume correlates to speed (in IOPS)
14+
default['ebs']['volumes']['/var/lib/jenkins']['dev'] = "/dev/sdj"
15+
default['ebs']['volumes']['/var/lib/jenkins']['fstype'] = "ext4"
16+
default['ebs']['volumes']['/var/lib/jenkins']['user'] = "jenkins"
17+
default['ebs']['volumes']['/var/lib/jenkins']['mountopts'] = 'noatime'
2018

21-
# To pin the jenkins version, must also override override['jenkins']['master']['source'] !!!
22-
# override['jenkins']['master']['version'] = '1.555'
23-
# override['jenkins']['master']['source'] = "#{node['jenkins']['master']['mirror']}/war/#{node['jenkins']['master']['version']}/jenkins.war"
24-
# override['jenkins']['master']['checksum'] = '31f5c2a3f7e843f7051253d640f07f7c24df5e9ec271de21e92dac0d7ca19431'
19+
default['ebs']['volumes']['/var/lib/artifactory']['size'] = 200 # size of the volume correlates to speed (in IOPS)
20+
default['ebs']['volumes']['/var/lib/artifactory']['dev'] = "/dev/sdk"
21+
default['ebs']['volumes']['/var/lib/artifactory']['fstype'] = "ext4"
22+
default['ebs']['volumes']['/var/lib/artifactory']['user'] = "artifactory"
23+
default['ebs']['volumes']['/var/lib/artifactory']['mountopts'] = 'noatime'
2524

26-
## GITHUB OAUTH
27-
default['master']['github']['webUri'] = 'https://github.com/'
28-
default['master']['github']['apiUri'] = 'https://api.github.com'
29-
default['master']['github']['adminUserNames'] = 'adriaanm,retronym,lrytz,chef,scala-jenkins'
30-
default['master']['github']['organizationNames'] = 'scala'
31-
default['master']['github']['useRepositoryPermissions'] = 'true'
32-
default['master']['github']['allowAnonymousReadPermission'] = 'true'
33-
default['master']['github']['authenticatedUserReadPermission'] = 'true'
34-
default['master']['github']['allowGithubWebHookPermission'] = 'true'
35-
default['master']['github']['allowCcTrayPermission'] = 'false'
36-
default['master']['github']['authenticatedUserCreateJobPermission'] = 'false'
25+
# JAVA
26+
default['java']['jdk_version'] = '7'
27+
default['java']['install_flavor'] = 'openjdk'
3728

38-
## CONTACT INFO
39-
default['master']['adminAddress'] = "[email protected]"
40-
default['master']['jenkinsHost'] = "scala-ci.typesafe.com" # duplicated because attributes can't refer to each other...
41-
default['master']['jenkinsUrl'] = "https://scala-ci.typesafe.com/"
42-
default['master']['jenkins']['notifyUrl'] = "http://scala-ci.typesafe.com:8888/jenkins"
29+
# ARTIFACTORY
30+
default['artifactory']['zip_url'] = 'http://dl.bintray.com/content/jfrog/artifactory/artifactory-3.6.0.zip?direct'
31+
default['artifactory']['zip_checksum'] = '72c375ab659d302da0b196349e152f3d799c3cada2f4d09f9399281a06d880e8'
32+
default['artifactory']['home'] = '/var/lib/artifactory'
33+
default['artifactory']['log_dir'] = '/var/lib/artifactory/logs'
34+
default['artifactory']['java']['xmx'] = '2g'
35+
default['artifactory']['java']['extra_opts'] = '-server'
36+
default['artifactory']['user'] = 'artifactory'
37+
default['artifactory']['proxyName'] = scalaCiHost
38+
default['artifactory']['proxyPort'] = scalaCiPort
39+
default['artifactory']['address'] = "localhost"
40+
default['artifactory']['port'] = 8282 # internal use over http
41+
default['artifactory']['install_java'] = false
4342

44-
## WORKER CONFIG
45-
default['repos']['private']['realm'] = "Artifactory Realm"
46-
default['repos']['private']['host'] = "private-repo.typesafe.com"
47-
default['repos']['private']['pr-snap'] = "http://private-repo.typesafe.com/typesafe/scala-pr-validation-snapshots/"
48-
default['repos']['private']['release-temp'] = "http://private-repo.typesafe.com/typesafe/scala-release-temp/"
43+
# JENKINS
44+
override['jenkins']['master']['install_method'] = 'war'
45+
override['jenkins']['master']['listen_address'] = '127.0.0.1' # external traffic must go through nginx
46+
override['jenkins']['master']['user'] = 'jenkins'
47+
override['jenkins']['master']['group'] = 'jenkins'
48+
override['jenkins']['master']['jvm_options'] = '-server -Xmx4G -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError' # -Dfile.encoding=UTF-8
4949

50-
default['s3']['downloads']['host'] = "downloads.typesafe.com.s3.amazonaws.com"
50+
# To pin the jenkins version, must also override override['jenkins']['master']['source'] !!!
51+
# override['jenkins']['master']['version'] = '1.555'
52+
# override['jenkins']['master']['source'] = "#{node['jenkins']['master']['mirror']}/war/#{node['jenkins']['master']['version']}/jenkins.war"
53+
# override['jenkins']['master']['checksum'] = '31f5c2a3f7e843f7051253d640f07f7c24df5e9ec271de21e92dac0d7ca19431'
54+
55+
## GITHUB OAUTH
56+
default['master']['github']['webUri'] = 'https://github.com/'
57+
default['master']['github']['apiUri'] = 'https://api.github.com'
58+
default['master']['github']['adminUserNames'] = 'adriaanm,retronym,lrytz,chef,scala-jenkins'
59+
default['master']['github']['organizationNames'] = 'scala'
60+
default['master']['github']['useRepositoryPermissions'] = 'true'
61+
default['master']['github']['allowAnonymousReadPermission'] = 'true'
62+
default['master']['github']['authenticatedUserReadPermission'] = 'true'
63+
default['master']['github']['allowGithubWebHookPermission'] = 'true'
64+
default['master']['github']['allowCcTrayPermission'] = 'false'
65+
default['master']['github']['authenticatedUserCreateJobPermission'] = 'false'
66+
67+
## CONTACT INFO
68+
default['master']['adminAddress'] = "[email protected]"
69+
default['master']['jenkinsHost'] = scalaCiHost
70+
default['master']['jenkinsUrl'] = "https://#{scalaCiHost}/"
71+
default['master']['jenkins']['notifyUrl'] = "http://#{scalaCiHost}:8888/jenkins" # scabot listens here
5172

52-
# see below (note that default['master']['env'] can only indirect through node -- workerJavaOpts is not in scope)
53-
workerJavaOpts = "-Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4"
54-
default['jenkinsEnv']['JAVA_OPTS'] = workerJavaOpts
55-
default['jenkinsEnv']['ANT_OPTS'] = workerJavaOpts
56-
default['jenkinsEnv']['MAVEN_OPTS'] = workerJavaOpts # doesn't technically need the -Dpartest one, but oh well
73+
# see below (note that default['master']['env'] can only indirect through node -- workerJavaOpts is not in scope)
74+
workerJavaOpts = "-Dfile.encoding=UTF-8 -server -XX:+AggressiveOpts -XX:+UseParNewGC -Xmx2G -Xss1M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=128M -Dpartest.threads=4"
75+
default['jenkinsEnv']['JAVA_OPTS'] = workerJavaOpts
76+
default['jenkinsEnv']['ANT_OPTS'] = workerJavaOpts
77+
default['jenkinsEnv']['MAVEN_OPTS'] = workerJavaOpts # doesn't technically need the -Dpartest one, but oh well
5778

58-
# NOTE: This is a string that represents a closure that closes over the worker node for which it computes the environment.
59-
# (by convention -- see `environment((eval node["master"]["env"])...` in _master-config-workers
60-
# Since we can't marshall closures, while attributes need to be sent from master to workers, we must encode them as something that can be shipped...
61-
default['master']['env'] = <<-'EOH'.gsub(/^ {2}/, '')
62-
lambda{| node | Chef::Node::ImmutableMash.new({
63-
"JAVA_HOME" => node['java']['java_home'], # we get the jre if we don't do this
64-
"JAVA_OPTS" => node['jenkinsEnv']['JAVA_OPTS'],
65-
"ANT_OPTS" => node['jenkinsEnv']['ANT_OPTS'],
66-
"MAVEN_OPTS" => node['jenkinsEnv']['MAVEN_OPTS'],
67-
"prRepoUrl" => node['repos']['private']['pr-snap'],
68-
"releaseTempRepoUrl" => node['repos']['private']['release-temp']
69-
})}
70-
EOH
79+
# NOTE: This is a string that represents a closure that closes over the worker node for which it computes the environment.
80+
# (by convention -- see `environment((eval node["master"]["env"])...` in _master-config-workers
81+
# Since we can't marshall closures, while attributes need to be sent from master to workers, we must encode them as something that can be shipped...
82+
default['master']['env'] = <<-'EOH'.gsub(/^ {2}/, '')
83+
lambda{| node | Chef::Node::ImmutableMash.new({
84+
"JAVA_HOME" => node['java']['java_home'], # we get the jre if we don't do this
85+
"JAVA_OPTS" => node['jenkinsEnv']['JAVA_OPTS'],
86+
"ANT_OPTS" => node['jenkinsEnv']['ANT_OPTS'],
87+
"MAVEN_OPTS" => node['jenkinsEnv']['MAVEN_OPTS'],
88+
"prRepoUrl" => node['repos']['private']['pr-snap'],
89+
"releaseTempRepoUrl" => node['repos']['private']['release-temp']
90+
})}
91+
EOH
7192

72-
## PLUGIN
73-
default['master']['ec2-start-stop']['url'] = 'https://dl.dropboxusercontent.com/u/12862572/ec2-start-stop.hpi'
93+
## PLUGIN
94+
default['master']['ec2-start-stop']['url'] = 'https://dl.dropboxusercontent.com/u/12862572/ec2-start-stop.hpi'
7495

75-
# SCABOT
76-
default['scabot']['jenkins']['user'] = "scala-jenkins"
77-
default['scabot']['github']['repo_user'] = "scala"
96+
# SCABOT
97+
default['scabot']['jenkins']['user'] = "scala-jenkins"
98+
default['scabot']['github']['repo_user'] = "scala"
99+
end

0 commit comments

Comments
 (0)