Building 50 Cloud Servers BASH/API Automation

So, I had a good friend of mine who is a cloud Mentor at Rackspace, reach out to me concerning an easier way of deploying cloud-images, without each time a cloud server is spun up, having to unroll the image into a CBS. His customer wanted to simply have a ‘primary master’ CBS volume, a template of their site if you will, the equivalent of a ‘golden image’, the only difference was it was a CBS Volume. So I set about making this work. It would still take a few hours, at least to provision 50 to 200 servers, but it was much faster than the alternatives. Here is how I did it. I actually have some ideas for how to improve this but I’ve not yet implemented it. That goody is to come in later scripts.

#!/bin/bash

USERNAME='mycloudusername'
APIKEY='mycloudapikey'
ACCOUNT_NUMBER=100101011
API_ENDPOINT="https://lon.blockstorage.api.rackspacecloud.com/v1/$ACCOUNT_NUMBER/volumes"
MASTER_CBS_VOL_ID="d8a67ad1-8037-46bc-8790-efca2cb6e5bd"


TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`


# Populate CBS
for i in `seq 1 2`;
do

echo "Generating CBS Clone #$i"
#curl -s -vvvv  \
-X POST "$API_ENDPOINT" \
-H "X-Auth-Token: $TOKEN"  \
-H "X-Project-Id: $ACCOUNT_NUMBER" \
-H "Accept: application/json"  \
-H "Content-Type: application/json" -d '{"volume": {"source_volid": "d8a67ad1-8037-46bc-8790-efca2cb6e5bd", "size": 50, "display_name": "win-'$i'", "volume_type": "SSD"}}'  | jq .volume.id | tr -d '"' >> cbs.created

done

echo "Giving CBS 2 hour grace time for 50 CBS clone"
#sleep 7200

echo "Listing all CBS Volume ID's created"
cat cbs.created
echo ""


# Populate Nova
count=1;
echo "Populating Nova servers with CBS disk"
while read n; do
Echo "Build Task $n Started:"
nova --insecure --os-username mycloudusername --os-auth-system=rackspace  --os-tenant-name 100110111 --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password myapikeygoeshere boot --flavor general1-1 --block-device-mapping vda="$n":::1 Auto-win-"$count"
((count=count+1))

done < cbs.created

# Move the cbs.created.old away
mv cbs.created cbs.created.old -f

Requirements are nova and jq.
https://stedolan.github.io/jq/
https://developer.rackspace.com/blog/getting-started-using-python-novaclient-to-manage-cloud-servers/

Updating Metadata and inside view on Cloud Servers and thru Nova API

So, a customer today was playing around with some metadata i.e.

“meta”: {
“rax-heat”: “dfdjh32j21-121c-411d-912c-77209ffc6642”
},

He understood setting the meta data key value pairs like;

nova meta platform-minion0-pp.gb-lon1.kubernetes.metroscales.io set foo=bar

and wanted to retrieve inside and outside of the VM. This is easy to do and can be done like

supernova lon show 812c7fed-ae3b-43ff-a0a1-0f07d52b795a | grep metadata
| metadata                            | {"rax-heat": "dfdjh32j21-121c-411d-912c-77209ffc6642", "foo": "bar", "rax_service_level_automation": "Complete"} |

or with nova

nova show 812c7fed-ae3b-43ff-a0a1-0f07d52b795a | grep metadata

Also it can be done inside the VM directly with xenstore-read

xenstore-read vm-data/user-metadata/build_config

Upgrading Xen Tools on Rackspace Cloud instance Virtual Machine

So, what with the first gen to next gen migrations ongoing, a lot of people may need to upgrade their xen server tools to the most recent version. Anyone who is running 5.5 or lower should upgrade to xs tools 6.2.0 pronto, it’s much more stable and fixes a lot of bugs that might exist in the earlier tool verisons. Here is how to do it.

But first ALWAYS TAKE A BACKUP IMAGE

Just in case. Remember installing XEN TOOLS can break your container if done incorrectly or if god hates you .

wget http://boot.rackspace.com/files/xentools/xs-tools-6.2.0.iso

mkdir -p tmp; mount -o loop xs-tools-6.2.0.iso tmp; cd tmp/Linux; ./install.sh; cd ../..; umount tmp

An excellent guide

https://support.rackspace.com/how-to/installing-xenserver-tools-on-next-generation-windows-cloud-servers/

Installing and using Swiftly with Rackspace Cloud Files

So you want a commandline tool for managing cloud files containers? Enter, swiftly. Now you’ve seen examples before with curl, fog and pyrax, but here’s a special one which is a commandline application.

# Upgrade pip just for hoots
pip install --upgrade pip

# Install swiftly
pip install swiftly

Now swiftly is installed, how about configuring it in your home directory

# put a .swiftly.conf in your home directory, if your root i.e. /root/.swiftly.
vi ~/.swiftly.conf

This is what the configuration should look like

[swiftly]                                                                       
auth_user = mycloudusername                                                        
auth_key = mycloudapikey                                  
auth_url = https://identity.api.rackspacecloud.com/v2.0                         
region = lon
bash-4.2 Tue Feb 16 15:04:34 pirax-test ~# swiftly get

current
export
exports
gallery
images
lb_153727_TESTING_Nov_2015
lb_153727_TESTING_Oct_2015
magento-stack-single-magento_server-gxlxbm7b6be2
meh
meh2
rackspace_orchestration_templates_store
scripts
testing
testing999
versions

This gives us the output of all of the cloud containers as shown above. Pretty cool. But what about placing files in a container?

swiftly put -i ~/myfile.txt CONTAINER/path/to/file/somefilenamethatsdifferent.txt

So If I wanted to upload to meh2 I would do an

swiftly put -i ~/mylocalfile.txt meh2/some/container/path/somefileiuploaded.txt

The destination file can be called mylocalfile.txt if you want but I want to illustrate the target name can be different to the local source name.

Stopping a process from crapping out without PIDcrap

So, this one comes up a lot too. So you wanna run a process, and you don’t want it to crap out, you don’t want PIDCrap or any other lunatic solution that simply doesn’t work 100%. Well, welcome to until.

I’ve been executing a ruby script that does some stuff with fog.

ruby my-fog-cloud-files-container-deleter-thingy.rb

but, it keeps crapping out with lots of errors

I figured crapout no more and nabbed this handy snippet, credit to good ole stackoverflow

until ruby my-fog-cloud-files-container-deleter-thingy.rb; do
    echo "Server 'myserver' crashed with exit code $?.  Respawning.." >&2
    sleep 1
done

Now when it craps out, it continues where it left off.. nice, simple, elegant.

I don’t know what kind of error handling swiftly and pyrax has available in built, but this is a nice way to do it. Theoretically this oneliner might be of use for turbolift as well as any other batch like job which might end prematurely before the code deploy finishes. I wish cloud init had something like this

Removing and purging all Rackspace Cloud Files from all containers

So, this one comes up a lot at work, and thanks to knowitnot.com I came accross this example using ruby and fog, its simple, dont runnaway!

Regex lets you define the exact things you want to delete, but be careful. I’m using .* , I’ve removed it and replaced it with some_regex, to make sure people don’t accidentally nuke their cloud files 😀

Install first

yum install rubygem-fog.noarch
#!/usr/bin/env ruby
# Author: Jason Barnett 

require 'fog'

def delete_file(container, file_num, max_tries)
  max_retries ||= max_tries
  try = 0
  puts "(#{file_num} of #{container.files.count}) Removing #{container.files[file_num].key}"
  begin
    container.files[file_num].destroy
  rescue Excon::Errors::NotFound, Excon::Errors::Timeout, Fog::Storage::Rackspace::NotFound => e
    if try == max_retries
      $stderr.puts e.message
    else
      try += 1
      puts "Retry \##{try}"
      retry
    end
  end
end

def equal_div(first, last, num_of_groups)
  total      = last - first
  group_size = total / num_of_groups + 1

  top    = first
  bottom = top + group_size
  blocks = 1.upto(num_of_groups).inject([]) do |result, x|
    bottom = last if bottom > last
    result << [ top, bottom ]

    top    += group_size + 1
    bottom =  top + group_size

    result
  end

  blocks
end

service = Fog::Storage.new({
    :provider             => 'Rackspace',               # Rackspace Fog provider
    :rackspace_username   => 'your_rackspace_username', # Your Rackspace Username
    :rackspace_api_key    => 'your_api_key',            # Your Rackspace API key
    :rackspace_region     => :ord,                      # Defaults to :dfw
    :connection_options   => {},                        # Optional
    :rackspace_servicenet => false                      # Optional, only use if you're the Rackspace Region Data Center
})

containers = service.directories.select do |s|
  s.key =~ /^some_regex/  # Only delete containers that match the regexp
end

TOT_THREADS = 4
threads     = []

containers.each do |container|
  puts
  puts "-----------------------------------------"
  puts "-- Removing _ALL_ objects from #{container.key}"
  puts "-----------------------------------------"
  puts

  #puts "container.files.count: #{container.files.count}"

  ## separates the number of files into equal groups to distribute to each thread
  mygroups = equal_div(0, container.files.count - 1, TOT_THREADS)

  0.upto(TOT_THREADS - 1) do |thread|
    threads << Thread.new([ container, mygroups[thread] ]) { |tObject|
      tObject[1][0].upto(tObject[1][1]) do |x|
        delete_file(tObject[0], x, 5)
      end
    }

  end
  threads.each { |aThread|  aThread.join }
  puts "Deleting #{container.key}"
  container.destroy
end

The script works, supporting multiple threads:

(8178 of 10000) Removing conv/2015/04/15/08/bdce78e4cab875e9e17eeeae051b1128.log.gz
(3183 of 10000) Removing conv/2015/03/17/18/0ea71bf7a834fc02b01c7f65c4eb23b0.log.gz
(5685 of 10000) Removing conv/2015/04/01/08/e4d6a7a2ee83d0be116cca6b1a92ad2a.log.gz
(682 of 10000) Removing conv/2015/02/26/07/163f33cf4e33c64139ab3bd7092a9478.log.gz
(8179 of 10000) Removing conv/2015/04/15/09/09ef6c4ecaa9341e76648b6e175db888.log.gz
(5686 of 10000) Removing conv/2015/04/01/09/23fa1dce145c8343efd8e0227fd41e35.log.gz
(3184 of 10000) Removing conv/2015/03/17/18/66cb5ac707c40777a1a78b1486e1c4f3.log.gz
(683 of 10000) Removing conv/2015/02/26/07/35f28368ed45b2fb7c7076c3b78eb008.log.gz
(5687 of 10000) Removing conv/2015/04/01/09/42882b705c2d7cfe5c726ddec3e457fc.log.gz
(3185 of 10000) Removing conv/2015/03/17/18/73dbb421db0093279138b6f69f246c06.log.gz
(8180 of 10000) Removing conv/2015/04/15/09/1a01e6b1779b76b5221b1c8c08398b00.log.gz
(684 of 10000) Removing conv/2015/02/26/07/44a59e91f6632383f18ff253e4404dba.log.gz
(3186 of 10000) Removing conv/2015/03/17/18/8f8ba5791ab0c1979b168517b229ef74.log.gz
(5688 of 10000) Removing conv/2015/04/01/09/8ff43c8a5de8a41ee9890686150c8c19.log.gz
(8181 of 10000) Removing conv/2015/04/15/09/28035bc7882fe36bb33da670e6c71ac0.log.gz
(685 of 10000) Removing conv/2015/02/26/07/7947d1b3f712d175281d33a6073531d9.log.gz
(8182 of 10000) Removing conv/2015/04/15/09/34a597377f63709a75c9fd6e8a68e073.log.gz
(3187 of 10000) Removing conv/2015/03/17/18/a92a58f166cc9bd1ff54a5f996bb776b.log.gz
(5689 of 10000) Removing conv/2015/04/01/09/9a728570208bcac32fd11f1581a57fbb.log.gz

List Cloud Networks using Rackspace API

In the previous chapter we learnt how to add networks using the API. It’s really simple, its basically a network and label placeholder. But what about viewing the networks we have after we’ve made some? This is pretty simple to confirm.

I have simplified the code a bit to make it easier to read.

#!/bin/sh

USERNAME='mycloudusername'
APIKEY='mycloudapikey'
ACCOUNT_NUMBER=10010101
API_ENDPOINT="https://lon.networks.api.rackspacecloud.com/v2.0/$ACCOUNT_NUMBER"

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

curl -i -X GET https://lon.networks.api.rackspacecloud.com/v2.0/networks -H "X-Auth-Token: $TOKEN" 

Output

# ./list-networks.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  5143  100  5028  100   115   4472    102  0:00:01  0:00:01 --:--:--  4477
HTTP/1.1 200 OK
Date: Fri, 12 Feb 2016 10:13:49 GMT
Via: 1.1 Repose (Repose/6.2.0.2)
Date: Fri, 12 Feb 2016 10:13:49 GMT
Content-Type: application/json; charset=UTF-8
Content-Length: 336
Server: Jetty(9.2.z-SNAPSHOT)

{"networks": [{"status": "ACTIVE", "subnets": [], "name": "Isolatednet", "admin_state_up": true, "tenant_id": "10010101", "shared": false, "id": "ae36972f-5cba-4327-8bff-15d8b05dc3ee"}], "networks_links": [{"href": "http://localhost:9696/v2.0/networks?marker=ae36972f-5cba-4327-8bff-15d8b05dc3ee&page_reverse=True", "rel": "previous"}]}

Pretty cool, but the format kind of sucks, I forgot to use python |-mjson.tool or jq to format the json output. Lets do that now by adding the line to the end of the curl -i line.

Now the output is nice:

{
    "networks": [
        {
            "admin_state_up": true,
            "id": "ae36972f-5cba-4327-8bff-15d8b05dc3ee",
            "name": "Isolatednet",
            "shared": false,
            "status": "ACTIVE",
            "subnets": [],
            "tenant_id": "10010101"
        }
    ],
    "networks_links": [
        {
            "href": "http://localhost:9696/v2.0/networks?marker=ae36972f-5cba-4327-8bff-15d8b05dc3ee&page_reverse=True",
            "rel": "previous"
        }
    ]
}

The complete code will look like;

#!/bin/sh

USERNAME='mycloudusername'
APIKEY='mycloudapikey'
ACCOUNT_NUMBER=10010101
API_ENDPOINT="https://lon.networks.api.rackspacecloud.com/v2.0/$ACCOUNT_NUMBER"

TOKEN=`curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{ "auth":{"RAX-KSKEY:apiKeyCredentials": { "username":"'$USERNAME'", "apiKey": "'$APIKEY'" }} }' -H "Content-type: application/json" |  python -mjson.tool | grep -A5 token | grep id | cut -d '"' -f4`

# with header no formatting
#curl -i -X GET https://lon.networks.api.rackspacecloud.com/v2.0/networks -H "X-Auth-Token: $TOKEN"
# without header with formatting
curl -X GET https://lon.networks.api.rackspacecloud.com/v2.0/networks -H "X-Auth-Token: $TOKEN" | python -mjson.tool