ESXi Virtual Machine Notes

These are notes I'm keeping on the VMs I create
under my installation of ESXi and their set-up.

Here is my finished ESXi/vSphere Client set-up:

Table of Contents

Application Server
Sun/Oracle JDK
Tomcat
Chef client
Add a Jenkins user
Database Server
MongoDB
Replica-set configuration
/etc/mongodb.conf
MongoDB shell commands
Database Server (auxiliary)
Jenkins VM for Continuous Deployment
Install openjdk-6-jdk
Install ant v1.8+
Install Git
Jenkins
Jenkins plug-ins
Jenkins user public key set-up
Set up sendmail
Chef Server
Chef server installation and set-up
Ancillary
Set up RSA keys for quicker access
Big-IP
Application Server
    1 February 2013

    This is an application server template. It consists of the following software
    components:

      - OpenSSL (installed as part of Ubuntu install)
      - Ubuntu Precise Pangolin 12.04 (LTS)
      - Sun/Oracle JDK 1.6.0-38
      - Sun/Oracle JCE Policy (two JARs integrated into the Sun JDK)
      - Tomcat 6.0.35
      - Chef client 10.18.2

    The versions indicated are what this virtual machine started out with.

    The following resources are included in this virtual machine:

      - 1 vCPU
      - 4Gb memory
      - 50Gb disk
      - IP address 192.168.0.29

Application server installation scrapes

Sun/Oracle JDK

If you do this, you'll have to install special JARs to extend the limited ones that come standard in compliance with U.S. export restrictions on cryptographic content software. The OpenJDK installed by Ubuntu does not observe such restrictions.

Tomcat

I suggest https://help.ubuntu.com/10.04/serverguide/tomcat.html no matter what version Ubuntu you're running. Here is a scrape from another box I installed.

    ~ $ ssh keltia.site
    Linux keltia 2.6.32-46-generic #105-Ubuntu SMP Fri Mar 1 00:08:49 UTC 2013 i686 GNU/Linux
    Ubuntu 10.04.4 LTS

    Welcome to Ubuntu!
     * Documentation:  https://help.ubuntu.com/

    New release 'precise' available.
    Run 'do-release-upgrade' to upgrade to it.

    Last login: Fri Mar 22 08:57:08 2013 from txn06fwb05-nat01.austin.hp.com
    root@keltia:~>  ps -ef | grep [t]omcat
    russ@keltia:~>  locate tomcat6
    russ@keltia:~>  sudo apt-get install tomcat6
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      linux-headers-2.6.32-21 ttf-dejavu-extra linux-headers-2.6.32-21-generic
      default-jre openjdk-6-jre icedtea-netx daemon
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      authbind
    Suggested packages:
      tomcat6-docs tomcat6-examples
    The following NEW packages will be installed:
      authbind tomcat6
    0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
    Need to get 49.7kB of archives.
    After this operation, 414kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ lucid-updates/main tomcat6 6.0.24-2ubuntu1.12 [32.8kB]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ lucid/main authbind 1.2.0build3 [16.9kB]
    Fetched 49.7kB in 0s (143kB/s)
    Selecting previously deselected package tomcat6.
    (Reading database ... 617801 files and directories currently installed.)
    Unpacking tomcat6 (from .../tomcat6_6.0.24-2ubuntu1.12_all.deb) ...
    Selecting previously deselected package authbind.
    Unpacking authbind (from .../authbind_1.2.0build3_i386.deb) ...
    Processing triggers for ureadahead ...
    Processing triggers for man-db ...
    Setting up tomcat6 (6.0.24-2ubuntu1.12) ...
    Adding system user `tomcat6' (UID 121) ...
    Adding new user `tomcat6' (UID 121) with group `tomcat6' ...
    Not creating home directory `/usr/share/tomcat6'.
     * Starting Tomcat servlet engine tomcat6                                [ OK ]

    Setting up authbind (1.2.0build3) ...

Afterwards, I launched a browser and typed http://keltia.site:8080 and saw Tomcat's default, "It works !" message.

Chef client

I browsed to http://www.opscode.com/chef/install and clicked the Chef Client tab. Selected Ubuntu, 12.04 (Precise) and x86_64. Then I followed the instructions thus:

    russ@app-1:~$ curl -L https://www.opscode.com/chef/install.sh | sudo bash
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  6510  100  6510    0     0   8541      0 --:--:-- --:--:-- --:--:-- 34444
    Downloading Chef  for ubuntu...
    Installing Chef
    Selecting previously unselected package chef.
    (Reading database ... 266182 files and directories currently installed.)
    Unpacking chef (from .../tmp.ReYwyLc5/chef__amd64.deb) ...
    Setting up chef (11.4.4-2.ubuntu.11.04) ...
    Thank you for installing Chef!

Add a Jenkins user

This isn't really Jenkins at all. It's a garden-variety user named jenkins for the purpose of copying (deploying) WARs to this application server. It could be any user, even my own, but then I'd be tying my user down to a corner case on my application host and I don't wish to do that. See Solution for multiple deployments by Jenkins.

(Notice here that I've created entries in all the /etc/hosts files as I set up the servers. I have two application server hosts, app-1 and app-2.)

    russ@app-1:~$ sudo bash
    root@app-1:~# useradd -d /home/jenkins -m jenkins
    root@app-1:~# passwd jenkins
    Enter new UNIX password: ********
    Retype new UNIX password: ********
    passwd: password updated successfully

(Not shown here: I'll be putting a public key in ~/.ssh for the real Jenkins user from a host running Jenkins. See this done when I set up Jenkins' below.)

In order to set up an RSA public key, I make this jenkins user a more canonical one. To do this, because of the lame way I created this user in the first place, I have fix /etc/passwd to make it use bash (instead of the Bourne shell). And I have to create the .ssh subdirectory.

    ~ $ ssh app-1
    Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-41-generic x86_64)
    .
    .
    .
    russ@app-1:~$ sudo bash
    root@app-1:~# vim /etc/passwd
    root@app-1:~# su - jenkins
    jenkins@app-1:~$ ll
    total 28
    drwxr-xr-x 3 jenkins jenkins 4096 May 16 07:01 ./
    drwxr-xr-x 4 root    root    4096 May 16 06:53 ../
    -rw-r--r-- 1 jenkins jenkins  220 Apr  3  2012 .bash_logout
    -rw-r--r-- 1 jenkins jenkins 3486 Apr  3  2012 .bashrc
    drwx------ 2 jenkins jenkins 4096 May 16 07:01 .cache/
    -rw-r--r-- 1 jenkins jenkins  675 Apr  3  2012 .profile
    -rw-r--r-- 1 jenkins jenkins  403 May 16 07:01 .ssh
    jenkins@app-1:~$ mkdir .ssh

Last, anticipating that Jenkins will be copying WAR files over to this server, I make Tomcat's webapps subdirectory writable:

    ~ $ ssh app-1
    Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-41-generic x86_64)
    .
    .
    .
    russ@app-1:~$ sudo bash
    root@app-1:~# cd /var/lib/tomcat6/
    root@app-1:/var/lib/tomcat6# ll
    total 24
    drwxr-xr-x  6 root    root    4096 Jan 26 20:56 ./
    drwxr-xr-x 36 root    root    4096 Jan 29 07:15 ../
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 common/
    lrwxrwxrwx  1 root    root      12 Apr 11  2012 conf -> /etc/tomcat6/
    lrwxrwxrwx  1 root    root      17 Apr 11  2012 logs -> ../../log/tomcat6/
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 server/
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 shared/
    drwxr-xr-x  4 tomcat6 tomcat6 4096 May 15 23:16 webapps/
    lrwxrwxrwx  1 root    root      19 Apr 11  2012 work -> ../../cache/tomcat6/
    root@app-1:/var/lib/tomcat6# chmod a+w webapps
    root@app-1:/var/lib/tomcat6# ll
    total 24
    drwxr-xr-x  6 root    root    4096 Jan 26 20:56 ./
    drwxr-xr-x 36 root    root    4096 Jan 29 07:15 ../
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 common/
    lrwxrwxrwx  1 root    root      12 Apr 11  2012 conf -> /etc/tomcat6/
    lrwxrwxrwx  1 root    root      17 Apr 11  2012 logs -> ../../log/tomcat6/
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 server/
    drwxr-xr-x  3 tomcat6 tomcat6 4096 Jan 26 20:56 shared/
    drwxrwxrwx  4 tomcat6 tomcat6 4096 May 15 23:16 webapps/
    lrwxrwxrwx  1 root    root      19 Apr 11  2012 work -> ../../cache/tomcat6/

Database Server
    1 February 2013

    This is a database server template. It consists of the following software
    components:

      - OpenSSL (installed as part of Ubuntu install)
      - Ubuntu Precise Pangolin 12.04 (LTS)
      - MongoDB 2.2.3
      - Chef client 10.18.2

    The versions indicated are what this virtual machine started out with.

    The following resources are included in this virtual machine:

      - 1 vCPU
      - 2Gb memory
      - 200Gb disk
      - IP address 192.168.0.30

Database server installation scrapes

MongoDB

    russ@db:~$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
    Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --secret-keyring /tmp/tmp.1XG7ZEgEtu --trustdb-name /etc/apt/trustdb.gpg --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --keyserver keyserver.ubuntu.com --recv 7F0CEB10
    gpg: requesting key 7F0CEB10 from hkp server keyserver.ubuntu.com
    gpg: key 7F0CEB10: public key "Richard Kreuter " imported
    gpg: no ultimately trusted keys found
    gpg: Total number processed: 1
    gpg:               imported: 1  (RSA: 1)

    russ@db:~$ sudo vim /etc/apt/sources.list.d/10gen.list
        deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen

    russ@db:~$ sudo apt-get update
        (much output...)

    russ@db:~$ sudo apt-get install mongodb-10gen
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following NEW packages will be installed:
      mongodb-10gen
    0 upgraded, 1 newly installed, 0 to remove and 80 not upgraded.
    Need to get 51.9 MB of archives.
    After this operation, 127 MB of additional disk space will be used.
    Get:1 http://downloads-distro.mongodb.org/repo/ubuntu-upstart/ dist/10gen mongodb-10gen amd64 2.2.3 [51.9 MB]
    Fetched 51.9 MB in 12s (4,114 kB/s)
    Selecting previously unselected package mongodb-10gen.
    (Reading database ... 48761 files and directories currently installed.)
    Unpacking mongodb-10gen (from .../mongodb-10gen_2.2.3_amd64.deb) ...
    Processing triggers for man-db ...
    Processing triggers for ureadahead ...
    ureadahead will be reprofiled on next reboot
    Setting up mongodb-10gen (2.2.3) ...
    Adding system user `mongodb' (UID 106) ...
    Adding new user `mongodb' (UID 106) with group `nogroup' ...
    Not creating home directory `/home/mongodb'.
    Adding group `mongodb' (GID 113) ...
    Done.
    Adding user `mongodb' to group `mongodb' ...
    Adding user mongodb to group mongodb
    Done.
    mongodb start/running, process 1525

Replica-set configuration

    9 April 2013

To set up a MongoDB replica set (we're not setting up sharding), there's only one file to modify on each node.

/etc/mongodb.conf:
    # mongodb.conf
    # --------------------------------------------------------------
    # These values are added here in configuration of a replica set.
    # No other values in this file were changed.
    port=37017*
    replSet=my-replicas
    fork=true
    # --------------------------------------------------------------
    .
    .
    .

* The port number is distinct on each node, 37017, 37018, etc.

Note that leaves the database path, dbpath, and logfile path, logpath at

    dpath=/var/lib/mongodb
    logpath=/var/log/mongodb

...respectively. Then, there are a series of commands to issue, which I did on the primary:

MongoDB shell commands for setting up a replica set

    config =
    {
        _id:"my-replicas", members:
        [
            { _id:0, host:"db-1:37017" },
            { _id:1, host:"db-2:37018" },
            { _id:2, host:"db-3:37019" }
        ]
    }

    rs.initiate( config )
    rs.conf()
    rs.status()

Console scrape of MongoDB shell commands executed from my development host (not an application or database node in my data center cluster):

    ~ $ mongo --host db1 --port 37017
    MongoDB shell version: 2.4.1
    connecting to: db1:37017/test
    > config =
    ... {
    ...     _id:"my-replicas", members:
    ...     [
    ...         { _id:0, host:"db-1:37017" },
    ...         { _id:1, host:"db-2:37018" },
    ...         { _id:2, host:"db-3:37019" }
    ...     ]
    ... }
    {
      "_id": "my-replicas",
      "members": [
        {
          "_id": 0,
          "host": "db-1:37017"
        },
        {
          "_id": 1,
          "host": "db-2:37018"
        },
        {
          "_id": 2,
          "host": "db-3:37019"
        }
      ]
    }
    > rs.initiate( config )
    {
      "info": "Config now saved locally.  Should come online in about a minute.",
      "ok": 1
    }
    > rs.conf()
    {
      "_id": "my-replicas",
      "version": 1,
      "members": [
        {
          "_id": 0,
          "host": "db-1:37017"
        },
        {
          "_id": 1,
          "host": "db-2:37018"
        },
        {
          "_id": 2,
          "host": "db-3:37019"
        }
      ]
    }
    my-replicas:PRIMARY> rs.status()
    {
      "set": "my-replicas",
      "date": ISODate("20130409T07:23:11-0600"),
      "myState": 1,
      "members": [
        {
          "_id": 0,
          "name": "db-1:37017",
          "health": 1,
          "state": 1,
          "stateStr": "PRIMARY",
          "uptime": 1037,
          "optime": {
            "t": 1365513704,
            "i": 1
          },
          "optimeDate": ISODate("20130409T07:21:44-0600"),
          "self": true
        },
        {
          "_id": 1,
          "name": "db-2:37018",
          "health": 1,
          "state": 2,
          "stateStr": "SECONDARY",
          "uptime": 83,
          "optime": {
            "t": 1365513704,
            "i": 1
          },
          "optimeDate": ISODate("20130409T07:21:44-0600"),
          "lastHeartbeat": ISODate("20130409T07:23:10-0600"),
          "pingMs": 0
        },
        {
          "_id": 2,
          "name": "db-3:37019",
          "health": 1,
          "state": 2,
          "stateStr": "SECONDARY",
          "uptime": 83,
          "optime": {
            "t": 1365513704,
            "i": 1
          },
          "optimeDate": ISODate("20130409T07:21:44-0600"),
          "lastHeartbeat": ISODate("20130409T07:23:10-0600"),
          "pingMs": 0
        }
      ],
      "ok": 1
    }
    my-replicas:PRIMARY> exit
    bye

Database Server (auxiliary)
    14 August 2013

    This is a database server template, for an auxiliary database node. It
	 consists of the following software components:

      - OpenSSL (installed as part of Ubuntu install)
      - Ubuntu Precise Pangolin 12.04 (LTS)
      - MongoDB 2.4.5
         - mongod, used as a MongoDB configuration server
         - mongos, used as a MongoDB sharding router
      - Chef client 10.18.2

    The versions indicated are what this virtual machine started out with.

    The following resources are included in this virtual machine:

      - 1 vCPU
      - 2Gb memory
      - 50Gb disk
      - IP address 192.168.0.48

Configuration

This node is not a member of a replica set. Instead, its function is to function as

  1. a shard-configuration server.
  2. a shard router.

This node will erect a single shard, with no shard key, that an application will contact in place of establishing its connection to the replica via a list of replica nodes. It also obviates the need to know the entire list and provide for it in /etc/hosts.

Instead, the application connects to this database node's hostname and port number. In turn, this node is connected to the three replica nodes (VMs) described in the previous section.

Database server installation scrapes

MongoDB

This proceeds identically as for the previous MongoDB installation.

Configuration server

    ? August 2013

To set up a MongoDB configuration server, we'll put MongoDB configuration in a different place from the usual /etc/mongodb.conf. Instead, it will be located at /data/mongodb/configsvr.conf.

/data/mongodb/configsvr.conf:
    # configsvr.conf
    # --------------------------------------------------------------
    # These values are added here in configuration of a replica set.
    # No other values in this file were changed.
    port=37017
    replSet=my-replicas
    fork=true
    # --------------------------------------------------------------
    .
    .
    .

Shard router

    ? August 2013

To set up a MongoDB sharding router, we'll put MongoDB configuration, again, in a different place from /etc/mongodb.conf, on the path /data/mongodb/sharding.conf.

/data/mongodb/sharding.conf:
    # sharding.conf
    # --------------------------------------------------------------
    # These values are added here in configuration of a replica set.
    # No other values in this file were changed.
    port=37017
    replSet=my-replicas
    fork=true
    # --------------------------------------------------------------
    .
    .
    .

Jenkins VM for Continuous Deployment

ESX details for a small, modest VM I set up for running pretty much only Jenkins. This illustration is to augment the Jenkins section and illustrates a vSphere step I did not show for any of the other VMs set up (though just as for this VM, I began each section with the details needed to do it.)

    19 February 2013

    This is a database server template. It consists of the following software
    components:

      - OpenSSL (installed as part of Ubuntu install)
      - Ubuntu Precise Pangolin 12.04 (LTS)
      - Jenkins
      - Chef client 10.18.2

    The versions indicated are what this virtual machine started out with.

    The following resources are included in this virtual machine:

      - 1 vCPU
      - 2Gb memory
      - 200Gb disk
      - IP address 192.168.0.30

Jenkins installation scrapes

Install openjdk-6-jdk

A JDK must be present given that Jenkins performs builds.

    root@jenkins:~# dpkg --list | grep jdk
    root@jenkins:~# apt-get install openjdk-6-jdk
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      linux-headers-3.2.0-29 linux-headers-3.2.0-37 linux-headers-3.2.0-29-generic
      linux-headers-3.2.0-37-generic
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      ca-certificates-java fontconfig fontconfig-config hicolor-icon-theme
      icedtea-6-jre-cacao icedtea-6-jre-jamvm icedtea-netx icedtea-netx-common
      java-common libasound2 libasyncns0 libatk-wrapper-java
    .
    .
    .
    Adding debian:RSA_Root_Certificate_1.pem
    done.
    Setting up openjdk-6-jre (6b27-1.12.3-0ubuntu1~12.04.1) ...
    update-alternatives: using /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/policytool to provide /usr/bin/policytool\
          (policytool) in auto mode.
    Setting up libatk-wrapper-java (0.30.4-0ubuntu2) ...
    .
    .
    .
    update-alternatives: using /usr/lib/jvm/java-6-openjdk-amd64/bin/xjc to provide /usr/bin/xjc (xjc) in auto mode.
    Setting up openjdk-6-jre-lib (6b27-1.12.3-0ubuntu1~12.04.1) ...
    Setting up libatk-wrapper-java-jni (0.30.4-0ubuntu2) ...
    Processing triggers for libc-bin ...
    ldconfig deferred processing now taking place

Install ant v1.8+

ant must be installed and it must be at least version 1.8.1.

    russ@jenkins:~$ which ant
    russ@jenkins:~$ sudo bash
    root@jenkins:~# which ant
    root@jenkins:~# su - jenkins
    jenkins@jenkins:~$ which ant
    russ@jenkins:~$ sudo apt-get install ant1.8
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    E: Unable to locate package ant1.8
    E: Couldn't find any package by regex 'ant1.8'

It turns out that Ubuntu doesn't think ant's very stable. The Eclipse team does. Here's how it's said you can best get it installed. What's going on here? The command to get special packages from other repositories (without specially adding those to /etc/apt/sources.list.d), i.e.: add-apt-repository, is a feature of Python software. That must be installed first. Even with this, there were disturbing respository update errors, but it did work.

    russ@jenkins:~$ sudo apt-get install python-software-properties
        (much output)
    russ@jenkins:~$ sudo add-apt-repository ppa:eclipse-team/debian-package
    You are about to add the following PPA to your system:
     PPA for the Eclipse package being developed by Debian.

    To install the OpenPGP key run
    sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com 5126890CDCC7AFE0
     More info: https://launchpad.net/~eclipse-team/+archive/debian-package
    Press [ENTER] to continue or ctrl-c to cancel adding it

    gpg: keyring `/tmp/tmp_QpyOz/secring.gpg' created
    gpg: keyring `/tmp/tmp_QpyOz/pubring.gpg' created
    gpg: requesting key DCC7AFE0 from hkp server keyserver.ubuntu.com
    gpg: /tmp/tmp_QpyOz/trustdb.gpg: trustdb created
    gpg: key DCC7AFE0: public key "Launchpad PPA for Eclipse Team" imported
    gpg: Total number processed: 1
    gpg:               imported: 1  (RSA: 1)
    OK
    russ@jenkins:~$ sudo apt-get update
    Ign http://ppa.launchpad.net precise InRelease
    Ign http://apt.opscode.com precise-0.10 InRelease
    .
    .
    .
    Fetched 2,984 kB in 3s (883 kB/s)
    W: Failed to fetch http://ppa.launchpad.net/eclipse-team/debian-package/ubuntu/dists/precise/main/source/Sources  404  Not Found
    W: Failed to fetch http://ppa.launchpad.net/eclipse-team/debian-package/ubuntu/dists/precise/main/binary-amd64/Packages  404  Not Found
    W: Failed to fetch http://ppa.launchpad.net/eclipse-team/debian-package/ubuntu/dists/precise/main/binary-i386/Packages  404  Not Found
    E: Some index files failed to download. They have been ignored, or old ones used instead.
    russ@jenkins:~$ sudo apt-get install ant
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      linux-headers-3.2.0-29 linux-headers-3.2.0-37 linux-headers-3.2.0-29-generic linux-headers-3.2.0-37-generic
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      ant-optional libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java
    Suggested packages:
      ant-gcj ant-doc libbsf-java liboro-java libxalan2-java libjaxp1.3-java junit liblog4j1.2-java libregexp-java
      jython antlr libbcel-java libcommons-logging-java libjdepend-java libgnumail-java libcommons-net-java libjsch-java
      javacc ant-optional-gcj libxerces2-java-doc libxerces2-java-gcj libxml-commons-resolver1.1-java-doc
    The following NEW packages will be installed:
      ant ant-optional libxerces2-java libxml-commons-external-java libxml-commons-resolver1.1-java
    0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
    Need to get 3,945 kB of archives.
    After this operation, 5,134 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main libxml-commons-resolver1.1-java all 1.2-7 [91.3 kB]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main libxml-commons-external-java all 1.4.01-2 [245 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main libxerces2-java all 2.11.0-4 [1,446 kB]
    Get:4 http://us.archive.ubuntu.com/ubuntu/ precise/main ant all 1.8.2-4build1 [1,837 kB]
    Get:5 http://us.archive.ubuntu.com/ubuntu/ precise/main ant-optional all 1.8.2-4build1 [326 kB]
    Fetched 3,945 kB in 3s (1,244 kB/s)
    Selecting previously unselected package libxml-commons-resolver1.1-java.
    (Reading database ... 111574 files and directories currently installed.)
    Unpacking libxml-commons-resolver1.1-java (from .../libxml-commons-resolver1.1-java_1.2-7_all.deb) ...
    Selecting previously unselected package libxml-commons-external-java.
    Unpacking libxml-commons-external-java (from .../libxml-commons-external-java_1.4.01-2_all.deb) ...
    Selecting previously unselected package libxerces2-java.
    Unpacking libxerces2-java (from .../libxerces2-java_2.11.0-4_all.deb) ...
    Selecting previously unselected package ant.
    Unpacking ant (from .../ant_1.8.2-4build1_all.deb) ...
    Selecting previously unselected package ant-optional.
    Unpacking ant-optional (from .../ant-optional_1.8.2-4build1_all.deb) ...
    russ@jenkins:~$ which ant
    /usr/bin/ant
    russ@jenkins:~$ ant -version
    Apache Ant(TM) version 1.8.2 compiled on December 3 2011
    russ@jenkins:~$ sudo su - jenkins
    jenkins@jenkins:~$ ant -version
    Apache Ant(TM) version 1.8.2 compiled on December 3 2011

Install Git

Because Jenkins is going to use Git, I must install it.

    russ@jenkins:~$ sudo apt-get install git
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      linux-headers-3.2.0-29 linux-headers-3.2.0-37 linux-headers-3.2.0-29-generic linux-headers-3.2.0-37-generic
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      git-man liberror-perl
    Suggested packages:
      git-daemon-run git-daemon-sysvinit git-doc git-el git-arch git-cvs git-svn git-email git-gui gitk gitweb
    The following NEW packages will be installed:
      git git-man liberror-perl
    0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
    Need to get 6,741 kB of archives.
    After this operation, 15.2 MB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main liberror-perl all 0.17-1 [23.8 kB]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main git-man all 1:1.7.9.5-1 [630 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main git amd64 1:1.7.9.5-1 [6,087 kB]
    Fetched 6,741 kB in 2s (2,700 kB/s)
    Selecting previously unselected package liberror-perl.
    (Reading database ... 112415 files and directories currently installed.)
    Unpacking liberror-perl (from .../liberror-perl_0.17-1_all.deb) ...
    Selecting previously unselected package git-man.
    Unpacking git-man (from .../git-man_1%3a1.7.9.5-1_all.deb) ...
    Selecting previously unselected package git.
    Unpacking git (from .../git_1%3a1.7.9.5-1_amd64.deb) ...
    Processing triggers for man-db ...
    Setting up liberror-perl (0.17-1) ...
    Setting up git-man (1:1.7.9.5-1) ...
    Setting up git (1:1.7.9.5-1) ...
    jenkins@jenkins:~$ git --version
    git version 1.7.9.5

Jenkins

Find Jenkins here: http://jenkins-ci.org/. There are download instructions for every platform at the following sequence of links:

  1. "Use Jenkins"
  2. "Using Jenkins", "Installing Jenkins"
  3. "Unix/Linux Installation", "Installing Jenkins on Ubuntu"
    russ@jenkins:~$ pwd
    /home/russ
    russ@jenkins:~$ wget -q -O - http://pkg.jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
    OK
    russ@jenkins:~$ sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian binary/ > /etc/apt/sources.list.d/jenkins.list'
    russ@jenkins:~$ sudo apt-get update
    Ign http://security.ubuntu.com precise-security InRelease
    Ign http://pkg.jenkins-ci.org binary/ InRelease
    Get:1 http://security.ubuntu.com precise-security Release.gpg [198 B]
    .
    .
    .
    Hit http://us.archive.ubuntu.com precise-backports/restricted Translation-en
    Get:50 http://us.archive.ubuntu.com precise-backports/universe Translation-en [16.2 kB]
    Fetched 3,003 kB in 3s (982 kB/s)
    Reading package lists... Done
    russ@jenkins:~$ sudo apt-get install jenkins
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    .
    .
    .
    The following extra packages will be installed:
      daemon
    The following NEW packages will be installed:
      daemon jenkins
    0 upgraded, 2 newly installed, 0 to remove and 53 not upgraded.
    Need to get 48.4 MB of archives.
    After this operation, 53.7 MB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe daemon amd64 0.6.4-1 [98.2 kB]
    Get:2 http://pkg.jenkins-ci.org/debian/ binary/ jenkins 1.504 [48.3 MB]
    Fetched 48.4 MB in 14s (3,427 kB/s)
    Selecting previously unselected package daemon.
    (Reading database ... 104576 files and directories currently installed.)
    Unpacking daemon (from .../daemon_0.6.4-1_amd64.deb) ...
    Selecting previously unselected package jenkins.
    Unpacking jenkins (from .../archives/jenkins_1.504_all.deb) ...
    Processing triggers for man-db ...
    Processing triggers for ureadahead ...
    Setting up daemon (0.6.4-1) ...
    Setting up jenkins (1.504) ...
    Adding system user `jenkins' (UID 106) ...
    Adding new user `jenkins' (UID 106) with group `nogroup' ...
    Not creating home directory `/var/lib/jenkins'.
     * Starting Jenkins Continuous Integration Server jenkins                [ OK ]

Chef client

See this done for other servers above.

Instant gratification

Above, at the end fo the Jenkins installation, you saw it launch Jenkins. It's accessed via http://jenkins:8080 because my development host has the following entry in /etc/hosts:

    192.168.0.34   jenkins.site   jenkins

Jenkins plug-ins

See A brief Exposé of Jenkins to see how to install plug-ins. I installed the following because they're relevant to my environment:

  1. Git Plugin Git Client Plugin GitHub Plugin GitHub pull request builder plugin

I also installed sendmail (see "Ancillary" below). I do not install the scp plug-in; it's useless.

Jenkins user public key set-up

On my application server hosts, I set up a user, jenkins, that I'll use from Jenkins here to copy any WAR files I wish to deploy. I have to a) create a public key (here), b) copy that key to those "Jenkins" users and c) ssh once from this host to each of those application server hosts to ensure that, later, Jenkins can invoke scripts to perform the copy with sufficient access.

Note below that the Jenkins installation seems to know already that I'm going to want to do this: it comes with a key already generated. I only need to copy that key to my application servers.

    ~ $ ssh jenkins
    Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-41-generic x86_64)
    .
    .
    .
    russ@jenkins:~$ sudo bash
    root@jenkins:~# su - jenkins
    jenkins@jenkins:~$ pwd
    /var/lib/jenkins
    jenkins@jenkins:~$ alias ll='ls -al'
    jenkins@jenkins:~$ ll .ssh
    total 16
    drwxr-xr-x  2 jenkins nogroup 4096 Mar  9 16:14 .
    drwxr-xr-x 11 jenkins adm     4096 May 10 18:14 ..
    -rw-------  1 jenkins nogroup 1679 Mar  9 16:14 id_rsa
    -rw-r--r--  1 jenkins nogroup  403 Mar  9 16:14 id_rsa.pub
    jenkins@jenkins:~$ cd .ssh
    jenkins@jenkins:~/.ssh$ scp id_rsa.pub jenkins@app-1:/home/jenkins/.ssh
    The authenticity of host 'app-1 (192.168.0.29)' can't be established.
    ECDSA key fingerprint is e7:71:f2:f7:ba:c8:98:d7:78:88:e1:46:dc:69:f1:b1.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'app-1,192.168.0.29' (ECDSA) to the list of known hosts.
    jenkins@app-1's password:
    id_rsa.pub                                    100%  403     0.4KB/s   00:00
    jenkins@jenkins:~/.ssh$ scp id_rsa.pub jenkins@app-2:/home/jenkins/.ssh
    The authenticity of host 'app-2 (192.168.0.33)' can't be established.
    ECDSA key fingerprint is d6:44:9c:ed:01:74:7a:7e:ae:30:94:2f:8f:3e:99:62.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'app-2,192.168.0.33' (ECDSA) to the list of known hosts.
    jenkins@app-2's password:
    id_rsa.pub                                    100%  403     0.4KB/s   00:00

Now that I've copied the keys over, I have to set them up to let Jenkins into the target hosts. Here's doing it on app-1:

    ~ $ ssh app-1
    Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-41-generic x86_64)
    .
    .
    .
    russ@app-1:~$ sudo bash
    root@app-1:~# su - jenkins
    jenkins@app-1:~$ ll
    total 28
    drwxr-xr-x 3 jenkins jenkins 4096 May 16 07:01 ./
    drwxr-xr-x 4 root    root    4096 May 16 06:53 ../
    -rw-r--r-- 1 jenkins jenkins  220 Apr  3  2012 .bash_logout
    -rw-r--r-- 1 jenkins jenkins 3486 Apr  3  2012 .bashrc
    drwx------ 2 jenkins jenkins 4096 May 16 07:01 .cache/
    -rw-r--r-- 1 jenkins jenkins  675 Apr  3  2012 .profile
    -rw-r--r-- 1 jenkins jenkins  403 May 16 07:01 .ssh
    jenkins@app-1:~$ cd .ssh
    jenkins@app-1:~/.ssh$ cat id_rsa.pub >> authorized_keys

Finally, I verify that the key works.

    jenkins@jenkins:~/.ssh$ ssh app-1
    Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-41-generic x86_64)
    .
    .
    .
    Last login: Thu May 16 07:07:19 2013 from 192.168.0.34
    jenkins@app-1:~$ logout
    Connection to app-1 closed.

And now, when Jenkins successfully builds my server application, it will be able to deploy it by copying it over to both application servers.

Set up sendmail on Jenkins server

This is so Jenkins can notify in case of broken build. Here's the full scrape for whatever worth that is.

    russ@jenkins:~$ sudo apt-get install sendmail
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      linux-headers-3.2.0-29 linux-headers-3.2.0-37 linux-headers-3.2.0-29-generic linux-headers-3.2.0-37-generic
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      libdb4.8 m4 procmail sendmail-base sendmail-bin sendmail-cf sensible-mda
    Suggested packages:
      sendmail-doc rmail logcheck sasl2-bin
    Recommended packages:
      default-mta mail-transport-agent fetchmail
    The following NEW packages will be installed:
      libdb4.8 m4 procmail sendmail sendmail-base sendmail-bin sendmail-cf sensible-mda
    0 upgraded, 8 newly installed, 0 to remove and 0 not upgraded.
    Need to get 2,098 kB of archives.
    After this operation, 7,548 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main libdb4.8 amd64 4.8.30-11ubuntu1 [679 kB]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main m4 amd64 1.4.16-2ubuntu1 [200 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main procmail amd64 3.22-19 [155 kB]
    Get:4 http://us.archive.ubuntu.com/ubuntu/ precise/universe sendmail-base all 8.14.4-2ubuntu2 [166 kB]
    Get:5 http://us.archive.ubuntu.com/ubuntu/ precise/universe sendmail-cf all 8.14.4-2ubuntu2 [102 kB]
    Get:6 http://us.archive.ubuntu.com/ubuntu/ precise/universe sendmail-bin amd64 8.14.4-2ubuntu2 [783 kB]
    Get:7 http://us.archive.ubuntu.com/ubuntu/ precise/universe sensible-mda amd64 8.14.4-2ubuntu2 [9,462 B]
    Get:8 http://us.archive.ubuntu.com/ubuntu/ precise/universe sendmail all 8.14.4-2ubuntu2 [3,470 B]
    Fetched 2,098 kB in 2s (755 kB/s)
    Selecting previously unselected package libdb4.8.
    (Reading database ... 111875 files and directories currently installed.)
    Unpacking libdb4.8 (from .../libdb4.8_4.8.30-11ubuntu1_amd64.deb) ...
    Selecting previously unselected package m4.
    Unpacking m4 (from .../m4_1.4.16-2ubuntu1_amd64.deb) ...
    Selecting previously unselected package procmail.
    Unpacking procmail (from .../procmail_3.22-19_amd64.deb) ...
    Selecting previously unselected package sendmail-base.
    Unpacking sendmail-base (from .../sendmail-base_8.14.4-2ubuntu2_all.deb) ...
    Selecting previously unselected package sendmail-cf.
    Unpacking sendmail-cf (from .../sendmail-cf_8.14.4-2ubuntu2_all.deb) ...
    Selecting previously unselected package sendmail-bin.
    Unpacking sendmail-bin (from .../sendmail-bin_8.14.4-2ubuntu2_amd64.deb) ...
    Selecting previously unselected package sensible-mda.
    Unpacking sensible-mda (from .../sensible-mda_8.14.4-2ubuntu2_amd64.deb) ...
    Selecting previously unselected package sendmail.
    Unpacking sendmail (from .../sendmail_8.14.4-2ubuntu2_all.deb) ...
    Processing triggers for install-info ...
    Processing triggers for man-db ...
    Processing triggers for ureadahead ...
    Setting up libdb4.8 (4.8.30-11ubuntu1) ...
    Setting up m4 (1.4.16-2ubuntu1) ...
    Setting up procmail (3.22-19) ...
    Setting up sendmail-base (8.14.4-2ubuntu2) ...
    adduser: Warning: The home directory `/var/lib/sendmail' does not belong to the user you are currently creating.
    Setting up sendmail-cf (8.14.4-2ubuntu2) ...
    Setting up sendmail-bin (8.14.4-2ubuntu2) ...
    update-rc.d: warning: sendmail stop runlevel arguments (0 1 6) do not match LSB Default-Stop values (1)
    update-alternatives: using /usr/lib/sm.bin/sendmail to provide /usr/sbin/sendmail-mta (sendmail-mta) in auto mode.
    update-alternatives: using /usr/lib/sm.bin/sendmail to provide /usr/sbin/sendmail-msp (sendmail-msp) in auto mode.
    update-alternatives: warning: not replacing /usr/share/man/man8/sendmail.8.gz with a link.

    You are doing a new install, or have erased /etc/mail/sendmail.mc.
    If you've accidentaly erased /etc/mail/sendmail.mc, check /var/backups.

    I am creating a safe, default sendmail.mc for you and you can
    run sendmailconfig later if you need to change the defaults.

     * Stopping Mail Transport Agent (MTA) sendmail                                                                [ OK ]
    Updating sendmail environment ...
    Validating configuration.
    Writing configuration to /etc/mail/sendmail.conf.
    Writing /etc/cron.d/sendmail.
    Could not open /etc/mail/databases(No such file or directory), creating it.
    Could not open /etc/mail/sendmail.mc(No such file or directory)
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Writing configuration to /etc/mail/sendmail.conf.
    Writing /etc/cron.d/sendmail.
    Turning off Host Status collection
    Could not open /etc/mail/databases(No such file or directory), creating it.
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Creating /etc/mail/databases...

    Checking filesystem, this may take some time - it will not hang!
      ...  Creating /etc/aliases (not found)
     Done.

    Checking for installed MDAs...
    Adding link for newly extant program (mail.local)
    Adding link for newly extant program (procmail)
    sasl2-bin not installed, not configuring sendmail support.

    To enable sendmail SASL2 support at a later date, invoke "/usr/share/sendmail/update_auth"


    Creating/Updating SSL(for TLS) information
    Creating /etc/mail/tls/starttls.m4...
    Creating SSL certificates for sendmail.
    Generating DSA parameters, 2048 bit long prime
    This could take some time
    .......+.....................+.+........+.................+.....+.....................++++[snip]
    ......+..+.+.......+....................................+..........+................+.....[snip]
    Generating RSA private key, 2048 bit long modulus
    .....................................+++
    ....+++
    e is 65537 (0x10001)

    *** *** *** WARNING *** WARNING *** WARNING *** WARNING *** *** ***

    Everything you need to support STARTTLS (encrypted mail transmission
    and user authentication via certificates) is installed and configured
    but is *NOT* being used.

    To enable sendmail to use STARTTLS, you need to:
    1) Add this line to /etc/mail/sendmail.mc and optionally
       to /etc/mail/submit.mc:
      include(`/etc/mail/tls/starttls.m4')dnl
    2) Run sendmailconfig
    3) Restart sendmail


    Updating /etc/hosts.allow, adding "sendmail: all".

    Please edit /etc/hosts.allow and check the rules location to
    make sure your security measures have not been overridden -
    it is common to move the sendmail:all line to the *end* of
    the file, so your more selective rules take precedence.
    Checking {sendmail,submit}.mc and related databases...
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Creating /etc/mail/databases...
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Creating /etc/mail/databases...
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Creating /etc/mail/Makefile...
    Reading configuration from /etc/mail/sendmail.conf.
    Validating configuration.
    Writing configuration to /etc/mail/sendmail.conf.
    Writing /etc/cron.d/sendmail.
    Disabling HOST statistics file(/var/lib/sendmail/host_status).
    Creating /etc/mail/sendmail.cf...
    Creating /etc/mail/submit.cf...
    Informational: confCR_FILE file empty: /etc/mail/relay-domains
    Warning: confCT_FILE source file not found: /etc/mail/trusted-users
     it was created
    Informational: confCT_FILE file empty: /etc/mail/trusted-users
    Warning: confCW_FILE source file not found: /etc/mail/local-host-names
     it was created
    Warning: access_db source file not found: /etc/mail/access
     it was created
    Updating /etc/mail/access...
    Linking /etc/aliases to /etc/mail/aliases
    Updating /etc/mail/aliases...
    WARNING: local host name (jenkins) is not qualified; see cf/README: WHO AM I?
    /etc/mail/aliases: 4 aliases, longest 10 bytes, 66 bytes total

    Warning: 3 database(s) sources
	    were not found, (but were created)
	    please investigate.
     * Starting Mail Transport Agent (MTA) sendmail                                        [ OK ]
    Setting up sensible-mda (8.14.4-2ubuntu2) ...
    Setting up sendmail (8.14.4-2ubuntu2) ...
    Processing triggers for libc-bin ...
    ldconfig deferred processing now taking place

Chef server

ESX details for a very small, modest VM I set up only for serving up Chef.

    31 May 2013

    This is a database server template. It consists of the following software
    components:

      - OpenSSL (installed as part of Ubuntu install)
      - Ubuntu Precise Pangolin 12.04 (LTS)
      - Chef server

    The versions indicated are what this virtual machine started out with.

    The following resources are included in this virtual machine:

      - 1 vCPU
      - 2Gb memory
      - 16Gb disk
      - IP address 192.168.0.24

Chef server installation and set-up

I browsed to http://www.opscode.com/chef/install and clicked the Chef Server tab. Selected Ubuntu, 12.04 (Precise) and x86_64. Then I selected the latest Chef version, 11.0.8, and clicked on the Debian download offered. I copied the Debian package to my Chef server and proceded as shown here:

    ~/Downloads/OS-downloads $ scp chef-server_11.0.8-1.ubuntu.12.04_amd64.deb chef:/home/russ
    chef-server_11.0.8-1.ubuntu.12.04_amd64.deb    100%  189MB  11.8MB/s   00:16
    ~/Downloads/OS-downloads $ ssh chef
    Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-generic x86_64)
    .
    .
    .
    russ@chef:~$ sudo dpkg -i chef-server_11.0.8-1.ubuntu.12.04_amd64.deb
    Selecting previously unselected package chef-server.
    (Reading database ... 48876 files and directories currently installed.)
    Unpacking chef-server (from chef-server_11.0.8-1.ubuntu.12.04_amd64.deb) ...
    Setting up chef-server (11.0.8-1.ubuntu.12.04) ...
    Thank you for installing Chef Server!

    The next step in the install process is to run:

    sudo chef-server-ctl reconfigure
    russ@chef:~$ sudo chef-server-ctl reconfigure
    Starting Chef Client, version 11.4.0
    Compiling Cookbooks...
    Recipe: chef-server::default
      * directory[/etc/chef-server] action create
        - create new directory /etc/chef-server
        - change mode from '' to '0775'
        - change owner from '' to 'root'
        - change group from '' to 'root'
    .
    .
    .
    Recipe: chef-server::erchef
      * service[erchef] action restart
        - restart service service[erchef]

    Chef Client finished, 270 resources updated
    chef-server Reconfigured!
    russ@chef:~$ df
    Filesystem            1K-blocks    Used Available Use% Mounted on
    /dev/mapper/chef-root  14356000 2003840  11631880  15% /
    udev                    1016768      12   1016756   1% /dev
    tmpfs                    410340     228    410112   1% /run
    none                       5120       0      5120   0% /run/lock
    none                    1025844       0   1025844   0% /run/shm
    /dev/sda1                233191   24992    195758  12% /boot
    russ@chef:~$ sudo reboot
    Broadcast message from russ@chef
	    (/dev/pts/0) at 21:29 ...

    The system is going down for reboot NOW!
    Connection to chef closed by remote host.
    Connection to chef closed.
    ~/Downloads/OS-downloads $ ssh chef
    Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-29-generic x86_64)
    .
    .
    .
    russ@chef:~$ ps -ef | grep [c]hef
    [lots of processes...]

Ancillary stuff

Set up RSA keys for quicker access

I made an RSA key, id_rsa.pub, which I copied to each and every VM, app-1.site, app-2.site, db-1.site, db-2.site, db-3.site and jenkins.site.

    $ scp id_rsa.pub app-1.site:/home/russ
    $ ssh app-1.site
    Password:
    $ mkdir .ssh
    $ mv id_rsa.pub .ssh/tuonela_rsa.pub
    $ cd .ssh
    $ cat tuonela_rsa.pub > authorized_keys
    $ logout
    $ ssh app-1.site
    (no password)

Big-IP

Since I can't afford Big-IP and someone told me Apache can be rigged to do the job of directing traffic to my two Tomcat servers on app-1 and app-2, I look into setting that up.

Here are some notes I'm putting together.