(Old) Chef Notes

With Chef, you can manage your servers by writing code, not by running commands; integrate tightly with your applications, databases, LDAP directories; easily configure applications that require knowledge about your entire infrastructure; and create perfect clones of QA environments, pre-production (staging) and production environments, etc.

If these notes appear schizophrenic, it's because they are. I'm trying to consume Chef at work and at home, the learning is somewhat painful because I'm not a Ruby guy at all. Someday, after this has come together, I'll try to consolidate and un-dummify these notes.

Note: I'm running Ubuntu Precise Pangolin in my examples. I'm using a subdirectory named chef in the root directory of my own development host as a place to work from. Names and passwords used in these notes are taken from great French chefs, including me
;-)


Links

Collection of links to open source Chef:


Setting up the Chef server and client is even easier (than below)

In the very recent past, this has gotten easier.

  1. Browse to http://www.opscode.com/chef/install.
  2. Under Installing the Chef client, select OS, version and platform architecture.
  3.  
    If behind a proxy or firewall, don't use the Quick Install, but Downloads to download the Chef client*:
     
  4. Choose desired Chef version.
  5. Copy downloaded package to host to install.

* Note that the Chef server install is always a download.


Setting up the Chef server

Nota bene: This method is deprecated and no longer the preferred approach.

Prepare Aptitude for the installation. Put the following line into /etc/apt/sources.list.d/opscode.list:

    deb http://apt.opscode.com/ precise-0.10 main

Continue on to get the GPG key. Couldn't get gpg keys because of the firewall. Had to do the following instead: From my personal host (not behind firewall), I get the opscode-keyring.gpg file, then copy it to my server:

    /home/russ/chef $ sudo mkdir -p /etc/apt/trusted.gpg.d
    /home/russ/chef $ cd /etc/apt/trusted.gpg.d/
    /etc/apt/trusted.gpg.d $ sudo  gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
    /etc/apt/trusted.gpg.d $ sudo gpg --export [email protected] \
             | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
    /etc/apt/trusted.gpg.d $ ll
    total 4
    drwxr-xr-x 2 root root   88 2012-10-25 11:25 .
    drwxr-xr-x 6 root root  400 2012-08-10 08:31 ..
    -rw-r--r-- 1 root root 1163 2012-10-25 11:25 opscode-keyring.gpg
    /etc/apt/trusted.gpg.d $ scp opscode-keyring.gpg [email protected]épin.acme.com:/home/russ
    opscode-keyring.gpg                                                                         100% 1163     1.1KB/s   00:00

Then, turning my attention back to my server (behind firewall):

    [email protected]épin:/etc/apt/trusted.gpg.d# ll
    total 16
    drwxr-xr-x 2 root root 4096 Oct 25 17:25 ./
    drwxr-xr-x 6 root root 4096 Oct  9 16:58 ../
    -rw-r--r-- 1 root root 1163 Oct 25 17:25 opscode-keyring.gpg
    [email protected]épin:~$ sudo apt-get update
    [sudo] password for russ:
    Ign http://pkg.jenkins-ci.org binary/ InRelease
    ...
    Hit http://us.archive.ubuntu.com precise-backports/universe Translation-en
    Fetched 2,208 kB in 10s (208 kB/s)
    Reading package lists... Done
    [email protected]épin:~$ sudo apt-get install opscode-keyring
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      libcommons-pool-java libecj-java authbind libtomcat6-java libcommons-dbcp-java libservlet2.5-java tomcat6-common...
    Use 'apt-get autoremove' to remove them.
    The following NEW packages will be installed:
      opscode-keyring
    0 upgraded, 1 newly installed, 0 to remove and 46 not upgraded.
    Need to get 2,584 B of archives.
    After this operation, 57.3 kB of additional disk space will be used.
    Get:1 http://apt.opscode.com/ precise-0.10/main opscode-keyring all 2010.11.26 [2,584 B]
    Fetched 2,584 B in 0s (8,068 B/s)
    Selecting previously unselected package opscode-keyring.
    (Reading database ... 118557 files and directories currently installed.)
    Unpacking opscode-keyring (from .../opscode-keyring_2010.11.26_all.deb) ...
    Setting up opscode-keyring (2010.11.26) ...

Upgrade the server software so that Chef's dependencies are the latest versions:

    [email protected]épin:~$ sudo apt-get upgrade
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages will be upgraded:
      accountsservice apparmor apport apport-symptoms apt apt-transport-https apt-utils build-essential coreutils dh-a...
    46 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
    Need to get 100 MB of archives.
    After this operation, 324 kB of additional disk space will be used.
    Do you want to continue [Y/n]? y
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main coreutils amd64 8.13-3ubuntu3.1 [2,216 kB]
    Get:2 http://pkg.jenkins-ci.org/debian/ binary/ jenkins 1.487 [49.0 MB]
    ...
    Processing triggers for resolvconf ...
    Processing triggers for initramfs-tools ...
    update-initramfs: Generating /boot/initrd.img-3.2.0-32-virtual

Now install Chef. Lots of stuff gets done here in terms of installing dependencies that aren't already on the host. Ruby is a good example of this.

    [email protected]épin:~$ sudo apt-get install chef chef-server
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages were automatically installed and are no longer required:
      authbind tomcat6-common
    Use 'apt-get autoremove' to remove them.
    The following extra packages will be installed:
      apache2-utils chef-expander chef-server-api chef-server-webui chef-solr couchdb couchdb-bin erlang-asn1 erlang-base ...
    ...

The URL for the Chef server must be established and also the admin/password for the web UI. This bit occurs as part of a TUI that's launched by the installation. I chose the following:

    http://chef.jacques-pépin.acme.com
    password for user chef in vhost "/chef":  Escoffier
    password for user admin in vhost "/chef": Escoffier

And, the installation continues from there unpacking lots more crap.

    ...
    Setting up libqtwebkit4 (2.2.1-1ubuntu4) ...
    Setting up libgecode27 (3.5.0-1~maverick1) ...
    Setting up libgecodegist30 (3.7.1-3) ...
    Setting up libgecodeflatzinc30 (3.7.1-3) ...
    Setting up libqt4-dev (4:4.8.1-0ubuntu4.2) ...
    update-alternatives: using /usr/bin/moc-qt4 to provide /usr/bin/moc (moc) in auto mode.
    update-alternatives: using /usr/bin/uic-qt4 to provide /usr/bin/uic (uic) in auto mode.
    update-alternatives: warning: skip creation of /usr/share/man/man1/uic.1.gz because associated file /usr/share/man/ma...
    Setting up libgecode-dev (3.7.1-3) ...
    Setting up libdep-selector-ruby (0.0.8-0opscode1) ...
    Setting up chef-server-api (10.16.0-1) ...

    Creating config file /etc/chef/server.rb with new version
     * Starting chef-server           ~ In 27728 [ OK ]
    Setting up chef-server (10.16.0-1) ...
    Setting up libqt4-opengl-dev (4:4.8.1-0ubuntu4.2) ...
    Setting up libqtwebkit-dev (2.2.1-1ubuntu4) ...
    Setting up chef-server-webui (10.16.0-1) ...

    Creating config file /etc/chef/webui.rb with new version
     * Starting chef-server-webui     ~ In 27914 [ OK ]
    Processing triggers for libc-bin ...
    ldconfig deferred processing now taking place

At this point, the documentation says it's a good idea to verify that six processes are running and gives the list. I did this very crudely:

    [email protected]épin:~$ ps -ef | grep [c]hef-server
    chef     27728     1  0 17:56 ?        00:00:01 merb : chef-server (api) : worker (port 4000)
    chef     27914     1  0 17:57 ?        00:00:00 merb : chef-server-webui : worker (port 4040)
    [email protected]épin:~$ ps -ef | grep [c]ouchdb
    couchdb  27493     1  0 17:56 ?        00:00:00 /bin/sh -e /usr/bin/couchdb -a /etc/couchdb/default.ini -a /etc/...
    couchdb  27502 27493  0 17:56 ?        00:00:00 /bin/sh -e /usr/bin/couchdb -a /etc/couchdb/default.ini -a /etc/...
    couchdb  27503 27502  0 17:56 ?        00:00:00 /usr/lib/erlang/erts-5.8.5/bin/beam -Bd -K true -A 4 -- -root /u...
    couchdb  27511 27503  0 17:56 ?        00:00:00 heart -pid 27503 -ht 11
    couchdb  27779 27503  0 17:56 ?        00:00:00 /usr/lib/couchdb/bin/couchjs /usr/share/couchdb/server/main.js
    [email protected]épin:~$ ps -ef | grep [r]abbit
    rabbitmq 26648     1  0 17:56 ?        00:00:00 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
    rabbitmq 26661     1  2 17:56 ?        00:00:05 /usr/lib/erlang/erts-5.8.5/bin/beam -W w -K true -A30 -P 1048576 ...
    rabbitmq 26747 26661  0 17:56 ?        00:00:00 /usr/lib/erlang/lib/os_mon-2.2.7/priv/bin/cpu_sup
    rabbitmq 26748 26661  0 17:56 ?        00:00:00 inet_gethost 4
    rabbitmq 26749 26748  0 17:56 ?        00:00:00 inet_gethost 4
    [email protected]épin:~$ ps -ef | grep [s]olr
    chef     27218     1  0 17:56 ?        00:00:01 java -Xmx256M -Xms256M -Dsolr.data.dir=/var/cache/chef/solr/data ...
    root     27343     1  0 17:56 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/chef-expander -d -c /etc/chef/solr.rb -...
    [email protected]épin:~$ ps -ef | grep [c]hef-expander
    root     27343     1  0 17:56 ?        00:00:00 /usr/bin/ruby1.8 /usr/bin/chef-expander -d -c /etc/chef/solr.rb -...
    root     27344 27343  1 17:56 ?        00:00:02 chef-expander worker #1 (vnodes 0-1023)

Configure the knife command line tool. I'm not certain yet why this would be done on the Chef server.

(Much more to come here...)


Setting up the Chef client

Install a Chef client on every node you wish to control. These would be VMs set up for this purpose.

The installation is identical to what went on above except when you reach the Aptitude command. If you're not operating from behind a firewall, you don't need to find someplace outside the firewall, but if you do, from a personal host with rights through the firewall:

    /home/russ/chef $ sudo mkdir -p /etc/apt/trusted.gpg.d
    /home/russ/chef $ cd /etc/apt/trusted.gpg.d/
    /etc/apt/trusted.gpg.d $ sudo  gpg --keyserver keys.gnupg.net --recv-keys 83EF826A

(In fact, this puts the keys on the path ~/.gnupg and they'll belong to root.) Continuing on...

    /etc/apt/trusted.gpg.d $ sudo gpg --export [email protected] \
             | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null
    /etc/apt/trusted.gpg.d $ scp opscode-keyring.gpg [email protected]:/home/russ

In the last bit, julia-childs.acme.com just happens to be the server where you want to copy the key-ring you had to get outside the firewall. If you did this directly on your server in the first place, the stuff's already there, but it's on the path /etc/apt/trusted.gpg.d/opscode-keyring.gpg. So, the next commands are unnecessary. (The "client node" is the server on which you're installing the Chef client as opposed to the host on which you have or will install the Chef server piece.)

From the client node:

    [email protected]:~$ sudo apt-get update
    [email protected]:~$ sudo apt-get install opscode-keyring

Now for the installation-proper which can be done because the key-ring's just where it needs to be. You're still on the host where you're installing the Chef client here. Note: if your host is a brand new VM in a data center and you've not upgraded it through Aptitude yet, you're going to see a lot of puzzling stuff go on. Just sit back and be patient.

    $ sudo apt-get upgrade

You'll be prompted for the Chef URL (where the Chef host will live) as discussed above. You may not want to specify it now, but save that for later if you don't yet have a Chef server set up.

    $ sudo apt-get install chef

Read the comment at the end of the next section on giving the path to the Chef server.


Setting up the Chef client on a data center VM

This relates more than just Chef. Here are scrapes and comments for installing Java, Tomcat and Chef client on a VM. Not covered here: setting up NTP (crucial).

    $ sudo apt-cache show sun-java
    N: Unable to locate package sun-java
    E: No packages found

--assumed Java not there. Basically, it's a mess to install Oracle Java; I'm going to use openjdk-6. If it's already there, this first command is not useful. The second command installs Tomcat 6.

    $ sudo apt-get install openjdk-6-jre-headless
    $ sudo apt-get install tomcat6 tomcat6-admin tomcat6-common tomcat6-user

Note: I actually installed a proper Sun JDK/JRE before hand, as well as Tomcat, so this wasn't a problem. I did find it necessary to install package tomcat6-user as it was not installed by the method linked here.

Now I'm ready to install the Chef client. I follow the instructions at:

http://wiki.opscode.com/display/chef/Installing+Chef+Client+on+Ubuntu+or+Debian

Be very careful! Do not do anything as root, only use sudo where shown here2 or you end up with a mess.

    $ sudo vim /etc/apt/sources.list.d/opscode.list

Place this line into that (new) file:

    deb http://apt.opscode.com/ precise-0.10 main

...because I'm installing on Ubuntu Precise Pangolin 12.04. Then, below, be really careful that you do NOT do the second two commands using sudo or you will sink yourself. I know because I tried it, then backed it out, because root can't do the second or then it can't do the install keyring.

    $ sudo mkdir -p /etc/apt/trusted.gpg.d
    $ gpg --keyserver keys.gnupg.net --recv-keys 83EF826A
    $ gpg --export [email protected] | sudo tee /etc/apt/trusted.gpg.d/opscode-keyring.gpg > /dev/null

You should get no errors updating Aptitude. If you do, you might not have put the right thing into opscode.list.

    $ sudo apt-get update
    $ sudo apt-get install opscode-keyring
    $ sudo apt-get upgrade

When I installed chef, I was careful NOT to supply a URL. Instead, I pressed Tab and then Enter (for OK). This is because a friend says not supplying one will make it so that the current VM can be converted to a more generic VM template, i.e.: you're doing the installation on a VM created with vSphere and are going to convert it to a template for creating later, definitive VMs.

    sudo apt-get install chef

There is nothing left to do (installing Ruby, etc.) if the VM is only going to run the Chef client.


The Chef server

The Chef server is not really a server at all. It doesn't do anything except publish. It stores data about nodes, roles, and user-provided data that's retrieved via essentially API calls. It provides a search API too.


The Chef client

Every time a Chef client runs, typically on one of the managed nodes, the following occurs:

  1. The node is built.
  2. Cookbooks are synchronized.
  3. The resource collection is compiled.
  4. The node is configured.
  5. Notices are given and exceptions handled.

The Chef node

The Chef node is a Ruby object that represents the host (or machine) that's being configured. It has attributes and a run list. The object is rebuilt every time the Chef client is run, merging input from the local host, the Chef API and attributes and run lists from roles.


The Chef cookbook

Cookbooks contain data including recipes, attributes and more. They are requested from the Chef server where they reside.


The resource collection

The resource collection is a list of resources that must be used to configure the node. This includes the results of evaluating recipes and attributes. Once resource collections are compiled, the required actions are taken by the providers and the node is saved back to the server where it's indexed for searching.


Chef Solo

Here is a tutorial on Chef Solo: http://www.youtube.com/watch?v=1G6bd4b91RU.

Chef Solo is where you don't necessarily install a server. Instead, you develop and test-deploy your recipes to your local development host. Later, you can use those recipes on any Chef server. On the Chef server, you invoke the chef-colo command and the server runs the recipes on all the nodes specified.


Step-by-step Chef configuration

Here are some instructions that assume that a Chef server is already set up and running, and that Chef clients are set up. The following configuration file is used in these steps. Prior to these steps, stuff is set up on the server (in its Chef repository) that's not shown here, but referred to in this file. Please see _________.

The Chef server URL below is the one you supplied during the installation of the Chef server.

knife.rb:
    log_level                :debug
    log_location             STDOUT
    node_name                'jack'
    client_key               '/home/jack/dev/chef-repos/.chef/jack.pem'
    chef_server_url          'http://16.86.192.111:4000/'
    cookbook_path          [ "/home/jack/dev/chef-repos/cookbooks",
                             "/home/jack/dev/chef-repos/site-cookbooks" ]
    cookbook_email           "[email protected]"
    cache_type               'BasicFile'
    cache_options( :path => '/home/jack/dev/chef-repos/checksums' )

Node registration

Do these steps, except as noted, on a client...

  1. Using NTP, synchronize time between clients and server. Chef will error out if the drift exceeds 15 minutes. (See Notes on NTP.)
  2. Get root.
  3. Create /etc/chef and make that the current working directory.
  4. Run knife configure client, knife.rb:
        # knife.rb ./
    

    This will generate client.rb and validation.pem. Edit client.rb and change log-level from "info" to "debug".

  5. Replace validation.pem with the one your Chef server as these must match perfectly.
  6. Run chef-client.
        # chef-client
    

    Unless everything is magically perfect, this will produce numerous errors that must be fixed. Frequently, these are errors in the configuration file, client.rb, such as a failure to find the server. In my observation, Chef will not make use of anything in /etc/hosts, so hard IP addresses must be used. I don't know whether it uses DNS as my set-ups have always been using ad hoc internal ESXi VMs or scrap hardware, never formal hosts noted in DNS tables.

  7. Now go to the Chef web interface to make sure that node is registered. You can do this from a browser running on your development host, which would not typically be the Chef server or any client. It's at this point that you'll make use of the password you supplied when installing the Chef server. Here, the username/password are (from the installation instructions earlier on this page), admin/Escoffier.
        http://16.86.192.111:4040
    

(NOTE THAT, STARTING HERE, NOTES ARE QUICK, ROUGH AND AS YET UNREFINED. PLEASE COME BACK LATER.)

Knife-up recipes

(Note that existing recipes already used on one client can be ferretted out using knife. Just find the look-up option for knife.)

These steps happen on any development host, except as noted, but not on the server or any client.

  1. Create a new subdirectory on your development host such as /home/jack/dev/chef-repos and create a hidden .chef directory. Place a copy of the knife.rb there. Change the properties to match that host.
  2. Go to the Chef web interface and create a user with admin rights. (See how to do this.). Copy the private key to a file named user.pem, e.g.: jack.pem. (Note: if you already had an admin user, and don't have access to the original PEM, i.e.: if you're inheriting maintenance of the Chef process, you'll have to re-create it and the original creator becomes invalid.)
  3. Create a cookbooks directory and add any cookbooks.
  4. Use knife commands to up-load the recipe (or recipes) which are Ruby files.
  5. In the web interface, verify that the cookbook is loaded correctly into the cookbook section.

Create a role

These steps happen in the web interface and on a chef client (node).

  1. Go to Roles and create a new role. In the menu assign it the cookbook from the steps above.
  2. Go to the web interface representation of the node being configured and add this role for it.
  3. Next go to the actual client node and run chef-client again.
  4. Verify that the changes have taken effect.

Creating a client

A client is...

To create a developer client, do this:

  1. Open the Chef web interface.
  2. Click the Clients tab.
  3. Click the Create tab.
  4. Fill in the new Name.
  5. Click Admin if appropriate.
  6. Copy (by scraping) and do not lose the private key presented.
  7. Create a PEM file to hold the key, probably on a path like /home/jack/dev/chef-repos/.chef/jack.pem.
  8. Ensure the PEM file is referenced from knife.rb.*

* This is the knife.rb placed usually on a path like /home/jack/dev/chef-repos/.chef. The path to the PEM file is indicated by client_key in knife.rb as shown here.

Errors

This is the most brittle part of using Chef. You will almost always get:

~/dev/chef-repos/.chef $ knife node list
ERROR: Failed to authenticate to http://16.86.192.111:4000/ as app02 with key /home/jack/dev/chef-repos/.chef/jack.pem
Response:  Failed to authenticate. Ensure that your client key is valid.

Hmmmm...


Retrieving an existing recipe

Recipes are managed (or can be) from an independent development host (rather than the Chef server or a client node).

Recipes are written in Ruby; filenames will end in .rb.

Let's imagine that a recipe already exists and is in use for one or more nodes. You've just taken over maintenance of your group's Chef installation and you want to use that recipe for another node that's not using it. To fetch an existing recipe down to your development host, do this:

  1. Launch a browser and point it at the Chef web interface, e.g.:
        http://16.86.192.111:4040
    
  2. Click the Cookbooks tab, the recipe should be in the list.
  3. Click the version number for the recipe.
  4. Under the recipe name, click the recipes link.
  5. Click on the Ruby file(s) associated with the recipe and scrape or otherwise save them to the appropriate place in filesystem, e.g.: /home/jack/dev/chef-repos.

Random notes as yet to coalesce

Chef controls nodes through recipes. Chef, when run, likes to create /etc/chef to do its work.

    $ knife configure client ./

—creates a certificate and Ruby file with configuration in it.

Go to http://hostname:4040, username: admin, password: Escoffier (set in TUI when Chef server installed).

Create a new client "installer" (what's this? "installer" was chosen, but any name could have been specified).

Role example: tomcat-role, recipes to configure Tomcat on a node.
db-role, recipes to configure, say, MongoDB on a node.
Etc.

To change Tomcat's server.xml, see cookbook/tomcat/template/default/server.xml.erb, default.rb.