Bootstrap jenkins from scratch -- the oy vey! edition

This functionality had bitrotted because we've never
really re-installed jenkins from scratch

With this PR, I'm able to bootstrap and fully configure
Jenkins ver. 1.611 on a ubuntu utopic vagrant box.

The provision has three phases: init, config, jenkins.

The config phase is needed when bootstrapping on EC2,
where chef vault is not yet available. After the instance
is up and authorized with chef-server for vault access,
we can go to the config phase.

Once Jenkins has the github oauth plugin and is restarted, we
can run the master-jenkins recipe.

NOTES:
 - switch to chef_vault_item for development in vagrant
 - Pass `hudson.model.User.allowNonExistentUserToLogin` to JVM
   to open jenkins up a bit for provisioning
 - Also need `org.eclipse.jetty.server.Request.maxFormContentSize=1000000`
   to allow plugin installs (??)

Author: adriaanm

.chef/Vagrantfile

@@ -6,11 +6,16 @@
 # centos: vagrant box add chef/centos-7.0
 Vagrant.configure("2") do |config|
   config.vm.box = "utopic-daily"
-  config.vm.provision :chef_solo do |chef|
-    chef.cookbooks_path = ["~/git/cookbooks"]
+  config.vm.provision :chef_zero do |chef|
+    chef.cookbooks_path = "~/git/cookbooks"
+    # after a long struggle tring to get chef-zero/solo to work with chef-vault, dumped the databags in plain text from admin account using:
+    # for item in $(knife data bag show master); do knife vault show --format json master $item  > .chef/data_bags/master/$item.json; done
+    chef.data_bags_path = "data_bags"
     chef.node_name = "jenkins-master"
+    # TODO: chef.encrypted_data_bag_secret_key_path
     chef.add_recipe("scala-jenkins-infra::master-init")
-    chef.add_recipe("scala-jenkins-infra::_master-config-proxy")
+    chef.add_recipe("scala-jenkins-infra::master-config")
+    chef.add_recipe("scala-jenkins-infra::master-jenkins") # do not run on first provision -- must run up to master-config and allow jenkins to restart
   end
   config.vm.network "public_network"
   config.vm.provider "virtualbox" do |v|

README.md

@@ -178,42 +178,64 @@ hub clone scala/scala-jenkins-infra
 cd scala-jenkins-infra
 ln -sh ~/git/cookbooks $PWD/.chef/
 
-knife site install cron
-knife site install logrotate
-knife site install chef_handler
-knife site install windows
-knife site install chef-client
-knife site install aws
-knife site install delayed_evaluator
-knife site install ebs
-knife site install java
-knife site install apt
-knife site install packagecloud
-knife site install runit
-knife site install yum
-knife site install jenkins
-knife site install 7-zip
-knife site install ark
-knife site install artifactory
-knife site install build-essential
-knife site install dmg
-knife site install yum-epel
-knife site install git
-knife site install user
-knife site install partial_search
-knife site install ssh_known_hosts
-knife site install git_user
-knife site install chef-sbt
-knife site install sbt-extras
-```
+knife cookbook site install cron
+knife cookbook site install logrotate
+knife cookbook site install chef_handler
+knife cookbook site install windows
+knife cookbook site install chef-client
+knife cookbook site install aws
+knife cookbook site install delayed_evaluator
+knife cookbook site install ebs
+knife cookbook site install apt
+knife cookbook site install packagecloud
+knife cookbook site install runit
+knife cookbook site install yum
+knife cookbook site install 7-zip
+knife cookbook site install ark
+knife cookbook site install artifactory
+knife cookbook site install build-essential
+knife cookbook site install dmg
+knife cookbook site install yum-epel
+knife cookbook site install git
+knife cookbook site install user
+knife cookbook site install partial_search
+knife cookbook site install ssh_known_hosts
+knife cookbook site install git_user
+
+knife cookbook site install chef-vault
+```
+
+### Current cookbooks
+ - 7-zip               ==  1.0.2
+ - apt                 ==  2.7.0
+ - ark                 ==  0.9.0
+ - artifactory         ==  0.1.1
+ - aws                 ==  2.7.0
+ - build-essential     ==  2.2.3
+ - chef-client         ==  4.3.0
+ - chef_handler        ==  1.1.6
+ - cron                ==  1.6.1
+ - delayed_evaluator   ==  0.2.0
+ - dmg                 ==  2.2.2
+ - ebs                 ==  0.3.6
+ - git                 ==  4.2.2
+ - git_user            ==  0.3.1
+ - logrotate           ==  1.9.1
+ - packagecloud        ==  0.0.17
+ - partial_search      ==  1.0.8
+ - runit               ==  1.6.0
+ - sbt                 ==  0.1.0
+ - sbt-extras          ==  0.4.0
+ - ssh_known_hosts     ==  2.0.0
+ - user                ==  0.4.2
+ - windows             ==  1.36.6
+ - yum                 ==  3.6.0
+ - yum-epel            ==  0.6.0
 
 ### Switch to unreleased versions from github
 ```
-//fixed: knife cookbook github install opscode-cookbooks/windows    # fix nosuchmethoderror (#150)
-//knife cookbook github install adriaanm/jenkins/fix305      # ssl fail on windows -- fix pending: https://github.com/opscode-cookbooks/jenkins/pull/313
-knife cookbook github install b-dean/jenkins/http_ca_fixes  # pending fix for above ^^^
-
-knife cookbook github install adriaanm/java/windows-jdk1.6 # jdk 1.6 installer barfs on re-install -- wipe its INSTALLDIR
+knife cookbook github install adriaanm/jenkins/fix305  # custom fixes + https://github.com/opscode-cookbooks/jenkins/pull/313 (b-dean/jenkins/http_ca_fixes)
+knife cookbook github install adriaanm/java/windows-jdk1.6  # jdk 1.6 installer barfs on re-install -- wipe its INSTALLDIR
 knife cookbook github install adriaanm/chef-sbt
 knife cookbook github install gildegoma/chef-sbt-extras
 knife cookbook github install adriaanm/artifactory

attributes/master.rb

@@ -18,7 +18,27 @@
   override['jenkins']['master']['listen_address'] = '127.0.0.1' # external traffic must go through nginx
   override['jenkins']['master']['user']           = 'jenkins'
   override['jenkins']['master']['group']          = 'jenkins'
-  override['jenkins']['master']['jvm_options']    = '-server -Xmx4G -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError' # -Dfile.encoding=UTF-8
+
+  # NOTES on override['jenkins']['master']['jvm_options']:
+  #  - org.eclipse.jetty.server.Request.maxFormContentSize is to fix:
+  #     WARNING: Caught exception evaluating: request.getParameter('q') in /updateCenter/byId/default/postBack. Reason: java.lang.IllegalStateException: Form too large 870330>500000
+
+  #  - hudson.model.User.allowNonExistentUserToLogin resolves issue with installing plugins
+  #    on bootstrapping jenkins (https://github.com/jenkinsci/jenkins/commit/80e9f3f50c3425c9b9b2bfdb58b03a1f1bd10aa3)
+  #   more of the stacktrace:
+  #      java.io.EOFException
+  #      	at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
+  #      before that:   #  ================================================================================
+  #      Error executing action `install` on resource 'jenkins_plugin[notification]'
+  #      ================================================================================
+  #
+  #      Mixlib::ShellOut::ShellCommandFailed
+  #      ------------------------------------
+  #      Expected process to exit with [0], but received '255'
+  #      ---- Begin output of "/usr/lib/jvm/java-7-openjdk-amd64/bin/java" -jar "/var/chef/cache/jenkins-cli.jar" -s http://localhost:8080 -i "/var/chef/cache/jenkins-key" install-plugin /var/chef/cache/notification-latest.plugin -name notification  ----
+
+  override['jenkins']['master']['jvm_options']    = '-server -Xmx4G -XX:MaxPermSize=512M -XX:+HeapDumpOnOutOfMemoryError -Dhudson.model.User.allowNonExistentUserToLogin=true -Dorg.eclipse.jetty.server.Request.maxFormContentSize=1000000' #
+  # -Dfile.encoding=UTF-8
 
   # To pin the jenkins version, must also override override['jenkins']['master']['source'] !!!
   # override['jenkins']['master']['version']  = '1.555'

files/default/jenkins.model.DownloadSettings.xml

@@ -0,0 +1,4 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<jenkins.model.DownloadSettings>
+  <useBrowser>true</useBrowser>
+</jenkins.model.DownloadSettings>

metadata.rb

@@ -6,6 +6,9 @@
 long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
 version          '0.1.0'
 
+# for chef_vault_item, which allows loading from plain databags when developing with vagrant
+depends 'chef-vault'
+
 depends 'chef-client'
 depends 'cron'
 

recipes/_master-config-jenkins.rb

@@ -0,0 +1,40 @@
+#
+# Cookbook Name:: scala-jenkins-infra
+# Recipe:: _master-config-jenkins
+#
+# Copyright 2014, Typesafe, Inc.
+#
+# All rights reserved - Do Not Redistribute
+#
+
+# Config base Jenkins setup and restart
+ruby_block 'set private key' do
+  block do
+    node.run_state[:jenkins_private_key] = chef_vault_item("master", "scala-jenkins-keypair")['private_key']
+  end
+end
+
+## set up chef user with public key from our master/scala-jenkins-keypair vault
+template "#{node['jenkins']['master']['home']}/users/chef/config.xml" do
+  source 'chef-user-config.erb'
+  user node['jenkins']['master']['user']
+  group node['jenkins']['master']['group']
+
+  variables({
+    :pubkey => chef_vault_item("master", "scala-jenkins-keypair")['public_key']
+  })
+end
+
+%w(notification ssh-credentials groovy cygpath job-dsl build-flow-plugin rebuild greenballs build-timeout copyartifact email-ext slack throttle-concurrents dashboard-view parameterized-trigger).each do |plugin|
+  plugin, version = plugin.split('=') # in case we decide to pin versions later
+  jenkins_plugin plugin
+end
+
+jenkins_plugin "ec2-start-stop" do
+  source   node['master']['ec2-start-stop']['url']
+  # doesn't support checksum
+end
+
+jenkins_plugin "github-oauth" do
+  notifies :restart, 'runit_service[jenkins]', :immediately
+end

recipes/_master-config-proxy.rb

@@ -35,10 +35,11 @@
   path "/etc/nginx/ssl/dhparam.pem"
 end
 
+
 file "/etc/nginx/ssl/scala-ci.key" do
   owner 'root'
   mode  '600'
-  content ChefVault::Item.load("master", "scala-ci-key")['private_key']
+  content chef_vault_item("master", "scala-ci-key")['private_key']
   sensitive true
 end
 

recipes/_master-config-scabot.rb

@@ -8,7 +8,7 @@
 #
 
 include_recipe "git"
-include_recipe "chef-sbt"
+include_recipe "sbt-extras"
 
 scabotHome     = "/home/scabot"
 scabotCheckout = "/home/scabot/scabot"
@@ -38,14 +38,15 @@
   owner     scabotUser
 end
 
+
 file "#{scabotHome}/.ssh/authorized_keys" do
   owner     scabotUser
   mode      '644'
-  content   ChefVault::Item.load("master", "scabot-keypair")['public_key']
+  content   chef_vault_item("master", "scabot-keypair")['public_key']
 end
 
-node.set['scabot']['github']['token']  = ChefVault::Item.load("master", "scabot")['github']['token']
-node.set['scabot']['jenkins']['token'] = ChefVault::Item.load("master", "scabot")['jenkins']['token']
+node.set['scabot']['github']['token']  = chef_vault_item("master", "scabot")['github']['token']
+node.set['scabot']['jenkins']['token'] = chef_vault_item("master", "scabot")['jenkins']['token']
 
 git_user scabotUser do
   home      scabotHome

recipes/_master-init-jenkins.rb

@@ -33,4 +33,13 @@
   source 'jenkins.model.JenkinsLocationConfiguration.xml.erb'
   user node['jenkins']['master']['user']
   group node['jenkins']['master']['group']
-end
+end
+
+# this is to allow plugin installation using jenkins-cli (needs to use REST to download update center json)
+cookbook_file "jenkins.model.DownloadSettings.xml" do
+  path "#{node['jenkins']['master']['home']}/jenkins.model.DownloadSettings.xml"
+  user  node['jenkins']['master']['user']
+  group node['jenkins']['master']['group']
+end
+
+

recipes/_master-config-auth-github.rb renamed to recipes/_master-jenkins-auth-github.rb

@@ -7,9 +7,7 @@
 # All rights reserved - Do Not Redistribute
 #
 
-require "chef-vault"
-
-apiVault = ChefVault::Item.load("master", "github-api")
+apiVault = chef_vault_item("master", "github-api")
 
 # This adds Github oAuth security. (login with your github id.)
 jenkins_script 'add_gh_authentication' do

recipes/_master-config-jobs.rb renamed to recipes/_master-jenkins-jobs.rb

@@ -7,11 +7,10 @@
 # All rights reserved - Do Not Redistribute
 #
 
-require "chef-vault"
 
 ruby_block 'set private key' do
   block do
-    node.run_state[:jenkins_private_key] = ChefVault::Item.load("master", "scala-jenkins-keypair")['private_key']
+    node.run_state[:jenkins_private_key] = chef_vault_item("master", "scala-jenkins-keypair")['private_key']
   end
 end
 

recipes/_master-config-workers.rb renamed to recipes/_master-jenkins-workers.rb

@@ -7,19 +7,18 @@
 # All rights reserved - Do Not Redistribute
 #
 
-require "chef-vault"
 
 ruby_block 'set private key' do
   block do
-    node.run_state[:jenkins_private_key] = ChefVault::Item.load("master", "scala-jenkins-keypair")['private_key']
+    node.run_state[:jenkins_private_key] = chef_vault_item("master", "scala-jenkins-keypair")['private_key']
   end
 end
 
 credentialsMap = {
   'jenkins'  => '954dd564-ce8c-43d1-bcc5-97abffc81c57'
 }
 
-privKey = ChefVault::Item.load("master", "scala-jenkins-keypair")['private_key']
+privKey = chef_vault_item("master", "scala-jenkins-keypair")['private_key']
 
 # TODO: different keypairs to sandbox different workers better, just in case?
 credentialsMap.each do |userName, uniqId|

recipes/_worker-config-debian.rb

@@ -31,14 +31,14 @@
       owner jenkinsUser
       mode '600'
       sensitive true
-      content ChefVault::Item.load("worker-publish", "chara-keypair")['private_key']
+      content chef_vault_item("worker-publish", "chara-keypair")['private_key']
     end
 
     execute 'accept chara host key' do
       command "ssh -oStrictHostKeyChecking=no [email protected] -i \"#{jenkinsHome}/.ssh/for_chara\" true"
       user jenkinsUser
       #
-      # not_if "grep -qs \"#{ChefVault::Item.load("worker-publish", "chara-keypair")['public_key']}\" #{jenkinsHome}/.ssh/known_hosts"
+      # not_if "grep -qs \"#{chef_vault_item("worker-publish", "chara-keypair")['public_key']}\" #{jenkinsHome}/.ssh/known_hosts"
     end
 
     directory "#{jenkinsHome}/.gnupg" do
@@ -50,7 +50,7 @@
         owner jenkinsUser
         mode '600'
         sensitive true
-        content Base64.decode64(ChefVault::Item.load("worker-publish", "gnupg")["#{kind}ring-base64"])
+        content Base64.decode64(chef_vault_item("worker-publish", "gnupg")["#{kind}ring-base64"])
       end
     end
 
@@ -70,12 +70,12 @@
         sensitive true
 
         variables({
-          :sonatypePass    => ChefVault::Item.load("worker-publish", "sonatype")['pass'],
-          :sonatypeUser    => ChefVault::Item.load("worker-publish", "sonatype")['user'],
-          :privateRepoPass => ChefVault::Item.load("worker-publish", "private-repo")['pass'],
-          :privateRepoUser => ChefVault::Item.load("worker-publish", "private-repo")['user'],
-          :s3DownloadsPass => ChefVault::Item.load("worker-publish", "s3-downloads")['pass'],
-          :s3DownloadsUser => ChefVault::Item.load("worker-publish", "s3-downloads")['user']
+          :sonatypePass    => chef_vault_item("worker-publish", "sonatype")['pass'],
+          :sonatypeUser    => chef_vault_item("worker-publish", "sonatype")['user'],
+          :privateRepoPass => chef_vault_item("worker-publish", "private-repo")['pass'],
+          :privateRepoUser => chef_vault_item("worker-publish", "private-repo")['user'],
+          :s3DownloadsPass => chef_vault_item("worker-publish", "s3-downloads")['pass'],
+          :s3DownloadsUser => chef_vault_item("worker-publish", "s3-downloads")['user']
         })
       end
     end

recipes/_worker-config-rhel.rb

@@ -7,6 +7,7 @@
 # All rights reserved - Do Not Redistribute
 #
 
+
 node["jenkinsHomes"].each do |jenkinsHome, workerConfig|
   jenkinsUser=workerConfig["jenkinsUser"]
 
@@ -24,8 +25,8 @@
       sensitive true
 
       variables({
-        :privateRepoPass => ChefVault::Item.load("worker", "private-repo-public-jobs")['pass'],
-        :privateRepoUser => ChefVault::Item.load("worker", "private-repo-public-jobs")['user']
+        :privateRepoPass => chef_vault_item("worker", "private-repo-public-jobs")['pass'],
+        :privateRepoUser => chef_vault_item("worker", "private-repo-public-jobs")['user']
       })
     end
   end

recipes/_worker-config-windows.rb

@@ -7,7 +7,6 @@
 # All rights reserved - Do Not Redistribute
 #
 
-require "chef-vault"
 
 node["jenkinsHomes"].each do |jenkinsHome, workerConfig|
   if workerConfig["publish"]
@@ -21,8 +20,8 @@
         user      workerConfig["jenkinsUser"]
 
         variables({
-          :s3DownloadsPass => ChefVault::Item.load("worker-publish", "s3-downloads")['pass'],
-          :s3DownloadsUser => ChefVault::Item.load("worker-publish", "s3-downloads")['user']
+          :s3DownloadsPass => chef_vault_item("worker-publish", "s3-downloads")['pass'],
+          :s3DownloadsUser => chef_vault_item("worker-publish", "s3-downloads")['user']
         })
       end
     end