Geekery of all stripes

Intro to Cloud Foundry and BOSH, part 2

· David Bishop

When last we were here, I was giving a broad overview of Cloud Foundry and BOSH, comparing them to Heroku and other PaaS’s. Today, we’re going to go over spinning up a Cloud Foundry instance from scratch, all on your handy local laptop. These instructions assume you are using OS X, though it should work for any platform that bosh-lite runs on. In fact, this tutorial closely follows the bosh-lite documentation, but is more tailored to beginners. Depending on the amount of bandwidth you have (downloading machine images can take a while) and the speed of your CPU (this does involve some package compilation, which can also take a while), this process might take up to an hour. I also assume that you have a working install of homebrew. It isn’t absolutely necessary, but it sure makes things easier.

Also, a point of clarification from my previous post (and I’ll go update that as well). During this process we’ll be using bosh-lite to provision Cloud Foundry on a single VM using containers. In production, with the “real” BOSH, we would instead be provisioning it with actual virtual machines. The fact that Cloud Foundry is running in containers is solely due to the fact that we’re using bosh-lite to give us a playground to test things. Keep that in mind as we go through this exercise.

First is the prerequisites. You need Virtual Box and Vagrant installed. As of this writing, Virtual Box is at 4.3.16 and Vagrant is at 1.6.5. You should be at those versions or better. You also need spiff

brew tap xoebus/homebrew-cloudfoundry
brew install spiff

and the Cloud Foundry cli

brew tap pivotal/tap
brew install cloudfoundry-cli

and finally the BOSH cli

gem install bosh_cli

Spiff is a tool that combines multiple BOSH manifests into one, which makes it easier to re-use components in BOSH. The Cloud Foundry CLI is what you will be using to push your final web app to your running Cloud Foundry instance. And the BOSH cli is the tool that allows you to interact with BOSH, running commands and connecting to virtual machines that are managed by BOSH.

Remember - the order is: virtual machine in VirtualBox that runs BOSH, a tool that helps orchestrate multiple virtual machines and/or containers at a time. We’ll be using BOSH to provision a cluster of containers running within that first machine, and that cluster is what will be running the Cloud Foundry software. Cloud Foundry is the software that makes it easy to spin up even more containers, which are the small slices of VMs that is all you need to run web applications. Refer back to this paragraph if you start to lose track of what the various relationships are.

Next, create a directory and clone the following two git repos into it:

Unless otherwise directed, run the rest of the commands in the top of the bosh-lite git repo.

Start the virtual machine that has been pre-configured to act as an “all-in-wonder” bosh director.

vagrant up --provider virtualbox

The first time you run that, vagrant will make you wait for a while as it fetches the bosh-lite-ubuntu-trusty box from the internet. Feel free to peruse twitter in the interim.

After bosh-lite is up and running in the vagrant VM, “login” to the bosh service by first targeting it and then logging in with the username/password combo of “admin/admin”.

bosh target lite
bosh login

That makes it so that when you run the BOSH cli commands, it knows which BOSH director to talk to. In the future when you’re a bosh wizard, controlling multiple BOSH directors with the flick of your wrists, you will run the ‘bosh targets’ command to list out all of the different instances that you’ve logged into, and switch between them using the alias that you’ve given them. In this case, we’ve aliased this BOSH director to ’lite’, so you’d switch to it with ‘bosh target lite’. For now, just know that’s how the bosh command knows where to go.

Before we move on, use ‘vagrant ssh’ to log into the bosh-lite VM. Take a poke around, that’s the machine that’s going to be hosting Cloud Foundry for you. You’ll see it’s running a bunch of different things: nginx, postgresql, resque, and some unusual things like ‘warden-linux’ and ‘bosh-agent’. All of those put together are “bosh”.

Next we’re going to use bosh to provision Cloud Foundry on your VM. From within the bosh-lite git repo (i.e., not within the Vagrant VM), run:


That script isn’t very long - 75 lines at this writing - but it’s going to do a lot. This is when bosh shines. It’s using the manifests from the cf-release git repo and combining them (using spiff) into one big manifest (take a look at bosh-lite/manifests/cf-manifest.yml to see it all put together) and then building out an infrastructure based on that. You’ll have plenty of time until bosh is done running, so take this time to go through the cf-manifest.yml file. You’ll see that’s it’s fairly logical (if long). It goes through and defines various instances, along with their IP address, disk space allocated, what template to use, etc. It also defines the networks and subnets, giving them a name and a range of IPs.

While you’re looking at that, BOSH is downloading all of the “stemcells” (VM templates) and building packages to use with those stemcells and produce functional machines. This part takes the longest of all, and heats up my laptop the most (16 minutes on a brand-new top-of-the-line Macbook Pro). I probably haven’t posted anything to twitter since you were downloading the vagrant instance, so instead go remind yourself that even though this is large and complicated, it’s not rocket science. Thank you to my friend Phil Seibel for that.

Now that Cloud Foundry has been provisioned, run

bosh vms

to see a list of all of the VMs it is using. It should look something like

| Job/index                          | State   | Resource Pool | IPs          |
| unknown/unknown                    | running | small_errand  | |
| unknown/unknown                    | running | small_errand  | |
| unknown/unknown                    | running | small_errand  | |
| api_z1/0                           | running | large_z1      | |
| etcd_z1/0                          | running | medium_z1     |  |
| ha_proxy_z1/0                      | running | router_z1     |  |
| hm9000_z1/0                        | running | medium_z1     | |
| loggregator_trafficcontroller_z1/0 | running | small_z1      |  |
| loggregator_z1/0                   | running | medium_z1     |  |
| login_z1/0                         | running | medium_z1     | |
| nats_z1/0                          | running | medium_z1     |   |
| postgres_z1/0                      | running | medium_z1     |  |
| router_z1/0                        | running | router_z1     |  |
| runner_z1/0                        | running | runner_z1     |  |
| uaa_z1/0                           | running | medium_z1     | |

If you ‘vagrant ssh’ back into the bosh-lite machine and look at the running processes, you’ll see each of those processes are children of various ‘wshd’ processes. ‘wshd’ is the warden daemon that I mentioned in my previous post, that forms the basis of Cloud Foundry and is akin to lxc or docker. For example, the ’etcd_z1/0’ container, when viewed from the bosh-lite machine:

root      8310  0.0  0.0   1120   240 ?        S<s  19:14   0:00 wshd: 4v0cs3g0arf
root      9062  0.0  0.0    188    32 ?        S<s  19:14   0:00  \_ runsvdir -P /etc/service log: rsyslogd: warning: ~ action is deprecated, consider using the 'stop' statement instead [try ] rsyslogd: action '*' treated as ':omusrmsg:*' - please change syntax, '*' will not be supported in the future [try ] rsyslogd: Could no open output pipe '/dev/xconsole': No such file or directory [try ] ........
root      9070  0.0  0.0    168     0 ?        S<s  19:14   0:00  |   \_ runsv ssh
root      9077  0.0  0.0  61364  2256 ?        S<   19:14   0:00  |   |   \_ /usr/sbin/sshd -D
root      9071  0.0  0.0    168     0 ?        S<s  19:14   0:00  |   \_ runsv agent
root      9087  0.0  0.0    184     4 ?        S<   19:14   0:00  |   |   \_ svlogd -tt /var/vcap/bosh/log
root      9088  0.0  0.1 349308  7412 ?        S<l  19:14   0:02  |   |   \_ /var/vcap/bosh/bin/bosh-agent -I warden -P ubuntu -C /var/vcap/bosh/agent.json
root      9072  0.0  0.0    168     0 ?        S<s  19:14   0:00  |   \_ runsv monit
root      9084  0.0  0.0    184     0 ?        S<   19:14   0:00  |   |   \_ svlogd -tt /var/vcap/monit/svlog
root      9351  0.0  0.0  91480  1548 ?        S<l  19:14   0:03  |   |   \_ /var/vcap/bosh/bin/monit -I -c /var/vcap/bosh/etc/monitrc
root      9073  0.0  0.0    168     0 ?        S<s  19:14   0:00  |   \_ runsv rsyslog
syslog   10815  0.0  0.0 205124  2456 ?        S<l  19:14   0:00  |       \_ rsyslogd -n -c5
vcap     10783  0.9  0.4 292512 26404 ?        S<l  19:14   1:11  \_ /var/vcap/packages/etcd/etcd -snapshot -data-dir=/var/vcap/store/etcd -addr= -peer-addr= -name=etcd_z1-0 -peer-heartbeat-timeout=50 -peer-election-timeout=1000
root     10785  0.0  0.0  17976   664 ?        S<   19:14   0:00  |   \_ /bin/bash /var/vcap/jobs/etcd/bin/etcd_ctl start
root     10787  0.0  0.0  17980   652 ?        S<   19:14   0:00  |   |   \_ /bin/bash /var/vcap/jobs/etcd/bin/etcd_ctl start
root     10804  0.0  0.0   4340   636 ?        S<   19:14   0:00  |   |   |   \_ logger -p -t vcap.etcd_ctl.stdout
root     10788  0.0  0.0   4348   584 ?        S<   19:14   0:00  |   |   \_ tee -a /dev/fd/63
root     10786  0.0  0.0  17976   660 ?        S<   19:14   0:00  |   \_ /bin/bash /var/vcap/jobs/etcd/bin/etcd_ctl start
root     10789  0.0  0.0  17984   656 ?        S<   19:14   0:00  |       \_ /bin/bash /var/vcap/jobs/etcd/bin/etcd_ctl start
root     10802  0.0  0.0   4340   624 ?        S<   19:14   0:00  |       |   \_ logger -p user.error -t vcap.etcd_ctl.stderr
root     10791  0.0  0.0   4348   584 ?        S<   19:14   0:00  |       \_ tee -a /dev/fd/63
vcap     10826  0.0  0.0 106472  4592 ?        S<l  19:14   0:00  \_ /var/vcap/packages/etcd_metrics_server/bin/etcd-metrics-server -index=0 -etcdAddress= -natsAddresses= -natsUsername=nats -natsPassword=nats -port=5678 -username= -password=
root     10828  0.0  0.0  17980   672 ?        S<   19:14   0:00      \_ /bin/bash /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl start
root     10830  0.0  0.0  17980   652 ?        S<   19:14   0:00      |   \_ /bin/bash /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl start
root     10844  0.0  0.0   4340   636 ?        S<   19:14   0:00      |   |   \_ logger -p -t vcap.etcd_metrics_server_ctl.stdout
root     10832  0.0  0.0   4348   580 ?        S<   19:14   0:00      |   \_ tee -a /dev/fd/63
root     10829  0.0  0.0  17976   664 ?        S<   19:14   0:00      \_ /bin/bash /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl start
root     10831  0.0  0.0  17984   660 ?        S<   19:14   0:00          \_ /bin/bash /var/vcap/jobs/etcd_metrics_server/bin/etcd_metrics_server_ctl start
root     10847  0.0  0.0   4340   636 ?        S<   19:14   0:00          |   \_ logger -p user.error -t vcap.etcd_metrics_server_ctl.stderr
root     10834  0.0  0.0   4348   580 ?        S<   19:14   0:00          \_ tee -a /dev/fd/63

However, each of those is separated out into their own containers. To ssh into a container, use ‘bosh ssh’:

bosh ssh etcd_z1/0

That will prompt for a password, use ‘admin’. Once you’re logged in, it should actually look exactly like the ps output I pasted above, except that there’s nothing else running (and the PIDs will be different - the wshd process will be PID 1, for instance).

Believe it or not, we’re just about ready to start pushing websites to your Cloud Foundry cluster (as long as they don’t require a database backend - that’s later). We’re going to create an “org” and “space” within CF for us to use.

cf api --skip-ssl-validation
cf auth admin admin
cf create-org me
cf target -o me
cf create-space development
cf target -s development

For the next bit, I’m going to cheat and send you elsewhere. I helped write some documentation on pushing your first ruby project to a CF instance, which you can find here. Scroll down to “Build a Ruby project” and follow the instructions exactly as given.

You’re back? Good. You should now have a small ruby application running in your test Cloud Foundry instance, happily telling the world “Hello!”. Interestingly, if you go ahead and run ‘bosh vms’ again to see your new, shiny ruby app, you’ll notice that there are no new containers provisioned. That’s because this new app is not being handled by bosh, and it isn’t running the bosh-agent. In fact, it’s container running within a container - the ‘runner_z1/0’ container. If you ‘bosh ssh runner_z1/0’ and run ‘ps axfuw’, you’ll notice that there is another wshd process with a child of ‘rackup’ underneath it (actually, bash then rackup, but who’s counting).

My next blog post will deal with setting up a database for your applications to use as a service. Until then, have fun doing things like ‘cf scale <appname> 3’ to see how easy it is to add more “droplets” (execution units, containers, whatever you want to call them). Also spend some time playing around with the cf and BOSH command-lines. How do you view logs from individual containers? What happens if you kill the wshd process of your sinatra example app from “the outside” (i.e., in the bosh-lite VM)? Kick the tires, it’s surprisingly difficult to break. If you do get in trouble, try ‘bosh cloudcheck’ (available as ‘bosh cck’ as well). And if worst comes to worst, you get to delete the Vagrant VM and start again, hardly the end of the world. And you’ll probably learn a little more each time you rebuild.

Happy Cloud Foundrying!

Places I blatantly stole commands from and whatnot:!