Cancelling a stuck soft reboot task on Xen Server

Today, one of my fellow colleagues received a call about a server that had run out of memory. They sent a soft reboot, and because of that the process task hung. This is because the hypervisor compute node sends a message to the nova agent running on the guest virtual machine! If the guest virtual machine has run out of memory, it’s not possible for nova to receive that command, or, if it does, then the soft (software) reboot can fail, because there is not enough memory to fork the process.

This could have been avoided by issuing a hard reboot straight away, but in this case we needed to cancel the task and send a hard reboot. Here is what I did:

List all pending tasks on xen-server

# xe task-list

uuid ( RO)                : a9f84f3d-0b96-8da2-a1d1-f5b774cd9173
          name-label ( RO): VM.clean_reboot
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.275

Cancel a pending task on xen-server

xe task-cancel uuid=a9f84f3d-0b96-8da2-a1d1-f5b774cd9173

This sets the active_state back to normal and gets rid of the ‘pending soft reboot’, but we need to restart the server too.

Using supernova API to stop and restart the server

supernova lon stop serveruuidhere
supernova lon start serveruuidhere

and…The customer is back up online and running, yay!

Using Rackspace Cloud Files & the API

Hi. So, when I started working at Rackspace I didn’t know very much about API. I know what it is, what it does, and why it’s used, but my experience was rather limited. So I was understandably a little bit concerned about using cloud files and API. Specifically using POST and GET thru CURL, and simple things such as authorization and identification thru header tokens.

First of all to use API, and make API requests we need an access token. The access token is totally different to my mycloud username, password and API Key itself. The token is a bit like a session, wheras the API key is a bit like a form username and password. It’s also worth noting, just like a http session, the $TOKEN will expire every now and then requiring you to authorise yourself to get a new token.

Authorisation & End Points

When you authorise yourself, you will get a token and a list of all the possible endpoints to GET and POST data, that is to say to retrieve or store records or query against particular search pattern, etc.

Step 1: Get User Token Thru Identity API using JSON auth structure

File: auth.json

{
    "auth": {
        "RAX-KSKEY:apiKeyCredentials": {
            "username": "mycloudusernamehere",
            "apiKey": "mycloudapikeyhere"
        }
    }
}
curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d @auth.json -H "Content-type: application/json"

It is possible to do this without the auth.json file and just use the string, like so:

 curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"RAX-KSKEY:apiKeyCredentials":{"username":"yourUserName","apiKey":"yourAPIPassword"}}}'  -H "Content-type: application/json" | python -m json.tool

It is also possible to connect to API using just USERNAME and API KEY:

curl https://identity.api.rackspacecloud.com/v2.0/tokens -X POST -d '{"auth":{"passwordCredentials":{"username":"yourUserName","password":"yourmycloudpassword"}}}' -H "Content-type: application/json"

After running one of these commands you will receive a large, and something like the below will be inside it:

This includes your full ‘token’ ID. In this case ‘BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw’

This is the token we need to use to authenticate against the API.

 

       "token": {
            "RAX-AUTH:authenticatedBy": [
                "APIKEY"
            ],
            "expires": "2015-09-29T17:04:56.092Z",
            "id": "BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw",
            "tenant": {
                "id": "1000000",
                "name": "1000000"
            }
        },

Step 2. Uploading a File with CURL to a Cloud File Container

# Set the TOKEN Variable in the BASH SHELL
TOKEN='BBBySDLKsxkj4CXdioidkj_a-vHqc4k6PYjxM2fu6D57Bf0dP0-Su6OO2beafdzoKDavyw32Sjd6SpiMhI-cUb654odmeiglz_2tsplnZ26T2Vj2h3LF-vwXNBEYS1IXvy7ZpARRMVranXw'

# CURL REQUEST TO API
CURL -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/1.txt -T /Users/adam9261/1.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/2.txt -T /Users/adam9261/2.txt -H "X-Auth-Token: $TOKEN"
curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/3.txt -T /Users/adam9261/3.txt -H "X-Auth-Token: $TOKEN"

Output

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:45 GMT
Content-Length: 0
Etag: 8aec10927922e92a963f6a1155ccc773
Content-Type: text/html; charset=UTF-8
X-Trans-Id: txca64c51affee434abaa79-0056097a34lon3
Date: Mon, 28 Sep 2015 17:34:45 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:46 GMT
Content-Length: 0
Etag: 83736eb7b9eb0dce9d11abcf711ca062
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx6df90d2dede54565bccfe-0056097a35lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

HTTP/1.1 100 Continue

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:34:47 GMT
Content-Length: 0
Etag: 42f2826fff018420731e7bead0f124df
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx79612b28754c4bb3aedd6-0056097a36lon3
Date: Mon, 28 Sep 2015 17:34:46 GMT

In this case I had 3 txt files, each approximately 1MB, that I uploaded to cloud files using CURL and the commandline. The pertinent part is “X-Auth-Token: $TOKEN”, which is the header which contains my authorisation key which was set at the commandline previously with TOKEN= line above it.

Now, it’s possible using a manifest to append these 3 files together into a single file. For instance if these files were larger, say 5GB and at the maximum limit for a file, such as a large dvd-iso, to exceed the 5GB limit we’d need to split that large file up into smaller files. But we want the cloud files container to send the whole file, all 3 5GB parts as if it were one file, this can be achieved with manifests. Here is how to do it!

Creating a Manifest file, defining 3 file parts as one single file download in the data stream

Step 1: Create Manifest File
In my case I created a manifest file first which tells cloud files which individual files make up the large file

[

        {
                "path": "/meh/1.txt",
                "etag": "8aec10927922e92a963f6a1155ccc773",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/2.txt",
                "etag": "83736eb7b9eb0dce9d11abcf711ca062",
                "size_bytes": 1048585
        },

        {
                "path": "/meh/3.txt",
                "etag": "42f2826fff018420731e7bead0f124df",
                "size_bytes": 1048587
        }
]

Step 2: Assign Manifest

curl -i -X PUT https://storage101.lon3.clouddrive.com/v1/MossoCloudFS_10045567/meh/mylargeappendedfile?multipart-manifest=put -d @manifest -H "X-Auth-Token: $TOKEN"

As you can see, it’s really quite simple, by giving the path, etag and size_bytes, it’s possible to append these 3 file references to a single file association. Now when downloading mylargeappendedfile from the cloud storage container, you’ll get /meh/1.txt /meh/2.txt and /meh/3.txt all appended to a single file. This is pretty handy when dealing with files over 5GB, as the maximum limit for an individual file is 5GB, but that doesn’t stop you from spreading some size across multiple files, just like for instance in a rar archive.

Generated Output:

HTTP/1.1 201 Created
Last-Modified: Mon, 28 Sep 2015 17:32:18 GMT
Content-Length: 0
Etag: "8f49c8861f0aef2eb750099223050c27"
Content-Type: text/html; charset=UTF-8
X-Trans-Id: tx64093603d9444e088f242-00560979a0lon3
Date: Mon, 28 Sep 2015 17:32:32 GMT

Compiling Grsecurity into a Linux Kernel

So, I am good friends with this really cool guy at work who is an excellent Linux Technician but also an extremely gifted pentester and security consultant. He has been telling me about the goodness of grsecurity and what it can do for my Linux Box. He says, even if my box is completely compromised, they probably won’t be able to do anything. Immediately after this I wanted to know what this rare moonshine was, and whether it was worth the trouble of kernel modifications and the whole shebang of configuration. After a day of on and off exploration at work, I have decided it is a most worthwhile endeavor and is probably the most extensive security you could install on a Linux server. That is, if you’re able to install it. For your average user it might be a stretch, so here is a nice little how to about how to achieve patching and compiling a linux Kernel with grsecurity module with PaX and advanced filesystem and kernel structure security. In other words, very darn cool.

For debian you might want to do something like
Step 1. Download Kernel Source (DEBIAN/possibly ubuntu)

apt-get source linux-image-$(uname -r)

Step 1. Download Kernel Source for http://www.kernel.org. In this case I’m compiling a version of 4.1.7

cd /tmp
wget https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.1.7.tar.gz

Step 2. Download the latest grsecurity kernel patch, being sure to match your grsecurity patch with the kernel version you want to use on your box

wget https://grsecurity.net/test/grsecurity-3.1-4.1.7-201509201149.patch

Step 3. untar the Linux Kernel, in my case I’m using Linux-4.1.7 Kernel

tar zxvf wget linux-4.1.7.tar.gz

Step 4. Apply the grsecurity patch in the linux-4.1.7 directory we just untarred into /tmp/linux-4.1.7

cd /tmp/linux-4.1.7
patch -p1 < ../grsecurity-3.1-4.1.7-201509201149.patch

Step 5. Ensure that the correct dependancies are installed for both compiling a kernel, and configuring the kernel

# needed for configuring a kernel with make menuconfig
apt-get install ncurses-dev

# needed for building a kernel with kpkg
apt-get install fakeroot kernel-package

Step 6. run make menuconfig within /tmp/linux-4.1.7

cd /tmp/linux-4.1.7
make menuconfig

Step 7. Refer to the grsecurity instructions on how to enable the grsecurity kernel module patches at https://grsecurity.net/quickstart.pdf

Navigate in the make menuconfig graphical interface as follows
Security Options -> GrSecurity --> *
Ensure that you are using Configuration Method (automatic), this is fine for most non power users. See the image below

Screen Shot 2015-09-24 at 5.23.26 PM

Step 8. Compile the Kernel Image, in debian this is something like;

fakeroot make-kpkg --initrd --revision=1 kernel_image

For other operating systems it will be more similar to

make dep bzImage modules modules_install install

For those persons that weren't able to complete this tutorial, maybe they will benefit from the documentation offered by grsecurity wiki, and the quickstart guide pdf they offer;

https://en.wikibooks.org/wiki/Grsecurity/Configuring_and_Installing_grsecurity
https://grsecurity.net/quickstart.pdf

Some more (very helpful) information about compiling kernel in debian:

https://www.debian.org/releases/stable/i386/ch08s06.html.en
https://debian-handbook.info/browse/stable/sect.kernel-compilation.html

Install and Configure Rackspace cloud backup Agent manually

Installing the cloud backup agent manually is quite easy. First take a look at http://meta.packages.cloudmonitoring.rackspace.com/ for your operating system distribution. In my case it’s Debian Wheezy (release 7).

From time to time the commands change, so always check the reference first (link above) before running the below command

wget http://meta.packages.cloudmonitoring.rackspace.com/debian-wheezy-x86_64/rackspace-cloud-monitoring-meta-stable_1.0_all.deb
dpkg -i rackspace-cloud-monitoring-meta-stable_1.0_all.deb
apt-get update
apt-get install rackspace-monitoring-agent

That’s it your done! Rackspace Cloud Backup agent is now manually running.

To be sure it is installed successfully and is starting up on boot, run a reboot and then verify in the rackspace control panel that it is running.

reboot

Screen Shot 2015-09-21 at 11.31.14 AM

If you at any point want to remove the monitoring agent, you can run:

apt-get remove rackspace-monitoring-agent

again confirming in the rackspace mycloud control panel that the service is no longer running:

Screen Shot 2015-09-21 at 11.34.27 AM

Automatically loading iptables in Debian and Ubuntu Linux

Debian doesn’t have an automatic way of loading up iptables, so if you use Debian, or Ubuntu you might be wondering why everytime you restart you lose your iptables rules. And if you got that far your probably wondering where the default iptables file is stored. Here is news, there is no file store! You have to make it! Here is how I did it.

1. Save your existing firewall rules to a file for later execution

iptables-save > /etc/firewall.conf 

2. Configure a firewall load command in if-up.d network startup script folder.

 
iptables-restore < /etc/firewall.conf

3. Make sure that the script you've put in the if up auto configuration is executable

 
chmod +x /etc/network/if-up.d/iptables

4. Now when you add new iptables rules you can run a save command

 
iptables-save > /etc/firewall.conf

or you can just edit the startup file, which will become active next reboot

vi /etc/firewall.conf 

This is one of the most elegant and simple ways to configure iptables, thanks to Major Hayden, a fellow Rackspace employee for this tip.

Viewing your servers processor details

Last week, I had an internal phone call, where a customer at their end of the line was asking for the specific hardware used in our openstack cloud solution. I told them that although there were no statistics available, and that most of the servers had a similar performance hardware configuration that the type of processors and their information can still be seen from the Guest virtual Machine. This means if the customer is running Linux they can run a command to return that information, and if they are running Windows they can run an application like CPU-Z to read the type of memory, processor etc being used.

Of course the command was simple

cat /proc/cpuinfo 

processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.12
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4521.97
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.02
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.03
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.04
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.00
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           L5520  @ 2.27GHz
stepping	: 5
cpu MHz		: 2261.060
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm syscall nx rdtscp lm constant_tsc ida nonstop_tsc pni monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr sse4_1 sse4_2 popcnt lahf_lm
bogomips	: 4522.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management: [8]

Booting a Virtual Machine using the Openstack API using supernova

It’s pretty damn simple to boot a virtual machine using the Openstack API. In this case I want to boot an image that is the same identical reference as the base image that a customer of ours is using that is getting some bad Cloud Block Storage performance. Being able to replicate the circumstances precisely is important when trying to recreate the symptoms of the customer. Configuring a new boot is really easy to do thru the API using supernova.

supernova customer boot testing-cbs-lon --image 9g46rb-79d6-45eb-9505-fb0bf31556d7 --flavor performance1-4

The above command is using my ‘customer’ auth user configuration (single API user, as opposed to admin API). I have asked API to boot a performance server with 4GB ‘performance1-4’ Using the image id ‘9g46rb-79d6-45eb-9505-fb0bf31556d7’ which is a customers image reference running windows. I tell the API I want to call the server testing-cbs-lon, and here is what I get back after running it

+--------------------------------------+-------------------------------------------------------------------+
| Property                             | Value                                                             |
+--------------------------------------+-------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                            |
| OS-EXT-STS:power_state               | 0                                                                 |
| OS-EXT-STS:task_state                | -                                                                 |
| OS-EXT-STS:vm_state                  | building                                                          |
| RAX-PUBLIC-IP-ZONE-ID:publicIPZoneId | e1cc4036fc52d50ca8ac2e90ff57e6b8c38d850bf6e3bd608e800971          |
| accessIPv4                           |                                                                   |
| accessIPv6                           |                                                                   |
| adminPass                            | censoredforsecurity                                               |
| config_drive                         |                                                                   |
| created                              | 2015-09-17T08:23:25Z                                              |
| flavor                               | 4 GB Performance (performance1-4)                                 |
| hostId                               |                                                                   |
| id                                   | d23cc039-ecc7-442a-81ad-b6bbb7a8c8e9                              |
| image                                | Windows Server 2008 R2 SP1 (8f39aeb0-79d6-45eb-9505-fb0bf31556d7) |
| key_name                             | -                                                                 |
| metadata                             | {}                                                                |
| name                                 | testing-cbs-lon                                                   |
| progress                             | 0                                                                 |
| status                               | BUILD                                                             |
| tenant_id                            | 10045567                                                          |
| updated                              | 2015-09-17T08:23:26Z                                              |
| user_id                              | 05b18e859cad42bb9a5a35ad0a6fba2f                                  |
+--------------------------------------+-------------------------------------------------------------------+

So it’s building, lets presume I want to check on the status of the server now. The id of the server, we can see is ‘d23cc039-ecc7-442a-81ad-b6bbb7a8c8e9’. So lets ask supernova to show the details for this server again for us.

supernova customer show d23cc039-ecc7-442a-81ad-b6bbb7a8c8e9
[SUPERNOVA] Running nova against customer...
+--------------------------------------+-------------------------------------------------------------------+
| Property                             | Value                                                             |
+--------------------------------------+-------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                            |
| OS-EXT-STS:power_state               | 0                                                                 |
| OS-EXT-STS:task_state                | spawning                                                          |
| OS-EXT-STS:vm_state                  | building                                                          |
| RAX-PUBLIC-IP-ZONE-ID:publicIPZoneId | e1cc4036fc52d50ca8ac2e90ff57e6b8c38d850bf6e3bd608e800971          |
| accessIPv4                           |                                                                   |
| accessIPv6                           |                                                                   |
| config_drive                         |                                                                   |
| created                              | 2015-09-17T08:23:25Z                                              |
| flavor                               | 4 GB Performance (performance1-4)                                 |
| hostId                               | d9a81c37f8f8aba51cd30a04151d99d2bdd60ad4504d4c95f4a3f609          |
| id                                   | d23cc039-ecc7-442a-81ad-b6bbb7a8c8e9                              |
| image                                | Windows Server 2008 R2 SP1 (8f39aeb0-79d6-45eb-9505-fb0bf31556d7) |
| key_name                             | -                                                                 |
| metadata                             | {}                                                                |
| name                                 | testing-cbs-lon                                                   |
| private network                      | 10.1.1.1                                                          |
| progress                             | 80                                                                |
| public network                       | 134.1.1.1     , 2a00:1a48:7810:101:be76:4eff:fe08:1f5d            |
| status                               | BUILD                                                             |
| tenant_id                            | 10045567                                                          |
| updated                              | 2015-09-17T08:28:28Z                                              |
| user_id                              | 05b18e859cad42bb9a5a35ad0a6fba2f                                  |
+--------------------------------------+-------------------------------------------------------------------+

As we can see the server is still building but it has started to get it’s network added.

If I wanted to see all the different flavors available (GB ram , disk, etc) we could run a

supernova customer flavor-list
[SUPERNOVA] Running nova against customer...
+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+
| ID               | Name                    | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |
+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+
| 2                | 512MB Standard Instance | 512       | 20   | 0         | 512  | 1     | 80.0        | N/A       |
| 3                | 1GB Standard Instance   | 1024      | 40   | 0         | 1024 | 1     | 120.0       | N/A       |
| 4                | 2GB Standard Instance   | 2048      | 80   | 0         | 2048 | 2     | 240.0       | N/A       |
| 5                | 4GB Standard Instance   | 4096      | 160  | 0         | 2048 | 2     | 400.0       | N/A       |
| 6                | 8GB Standard Instance   | 8192      | 320  | 0         | 2048 | 4     | 600.0       | N/A       |
| 7                | 15GB Standard Instance  | 15360     | 620  | 0         | 2048 | 6     | 800.0       | N/A       |
| 8                | 30GB Standard Instance  | 30720     | 1200 | 0         | 2048 | 8     | 1200.0      | N/A       |
| compute1-15      | 15 GB Compute v1        | 15360     | 0    | 0         |      | 8     | 1250.0      | N/A       |
| compute1-30      | 30 GB Compute v1        | 30720     | 0    | 0         |      | 16    | 2500.0      | N/A       |
| compute1-4       | 3.75 GB Compute v1      | 3840      | 0    | 0         |      | 2     | 312.5       | N/A       |
| compute1-60      | 60 GB Compute v1        | 61440     | 0    | 0         |      | 32    | 5000.0      | N/A       |
| compute1-8       | 7.5 GB Compute v1       | 7680      | 0    | 0         |      | 4     | 625.0       | N/A       |
| general1-1       | 1 GB General Purpose v1 | 1024      | 20   | 0         |      | 1     | 200.0       | N/A       |
| general1-2       | 2 GB General Purpose v1 | 2048      | 40   | 0         |      | 2     | 400.0       | N/A       |
| general1-4       | 4 GB General Purpose v1 | 4096      | 80   | 0         |      | 4     | 800.0       | N/A       |
| general1-8       | 8 GB General Purpose v1 | 8192      | 160  | 0         |      | 8     | 1600.0      | N/A       |
| io1-120          | 120 GB I/O v1           | 122880    | 40   | 1200      |      | 32    | 10000.0     | N/A       |
| io1-15           | 15 GB I/O v1            | 15360     | 40   | 150       |      | 4     | 1250.0      | N/A       |
| io1-30           | 30 GB I/O v1            | 30720     | 40   | 300       |      | 8     | 2500.0      | N/A       |
| io1-60           | 60 GB I/O v1            | 61440     | 40   | 600       |      | 16    | 5000.0      | N/A       |
| io1-90           | 90 GB I/O v1            | 92160     | 40   | 900       |      | 24    | 7500.0      | N/A       |
| memory1-120      | 120 GB Memory v1        | 122880    | 0    | 0         |      | 16    | 5000.0      | N/A       |
| memory1-15       | 15 GB Memory v1         | 15360     | 0    | 0         |      | 2     | 625.0       | N/A       |
| memory1-240      | 240 GB Memory v1        | 245760    | 0    | 0         |      | 32    | 10000.0     | N/A       |
| memory1-30       | 30 GB Memory v1         | 30720     | 0    | 0         |      | 4     | 1250.0      | N/A       |
| memory1-60       | 60 GB Memory v1         | 61440     | 0    | 0         |      | 8     | 2500.0      | N/A       |
| performance1-1   | 1 GB Performance        | 1024      | 20   | 0         |      | 1     | 200.0       | N/A       |
| performance1-2   | 2 GB Performance        | 2048      | 40   | 20        |      | 2     | 400.0       | N/A       |
| performance1-4   | 4 GB Performance        | 4096      | 40   | 40        |      | 4     | 800.0       | N/A       |
| performance1-8   | 8 GB Performance        | 8192      | 40   | 80        |      | 8     | 1600.0      | N/A       |
| performance2-120 | 120 GB Performance      | 122880    | 40   | 1200      |      | 32    | 10000.0     | N/A       |
| performance2-15  | 15 GB Performance       | 15360     | 40   | 150       |      | 4     | 1250.0      | N/A       |
| performance2-30  | 30 GB Performance       | 30720     | 40   | 300       |      | 8     | 2500.0      | N/A       |
| performance2-60  | 60 GB Performance       | 61440     | 40   | 600       |      | 16    | 5000.0      | N/A       |
| performance2-90  | 90 GB Performance       | 92160     | 40   | 900       |      | 24    | 7500.0      | N/A       |
+------------------+-------------------------+-----------+------+-----------+------+-------+-------------+-----------+

On the left are the names to use for the first command we entered. So we can spin up a whole host of different servers. If we wanted to know details about a specific flavor, then we can run a flavor-show command like so:

 
[SUPERNOVA] Running nova against customer...
+-----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                          | Value                                                                                                                                                            |
+-----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-EXT-DATA:ephemeral         | 900                                                                                                                                                              |
| OS-FLV-WITH-EXT-SPECS:extra_specs | {"number_of_data_disks": "3", "resize_policy_class": "performance_flavor", "class": "performance2", "disk_io_index": "70", "policy_class": "performance_flavor"} |
| disk                              | 40                                                                                                                                                               |
| extra_specs                       | {"number_of_data_disks": "3", "resize_policy_class": "performance_flavor", "class": "performance2", "disk_io_index": "70", "policy_class": "performance_flavor"} |
| id                                | performance2-90                                                                                                                                                  |
| name                              | 90 GB Performance                                                                                                                                                |
| ram                               | 92160                                                                                                                                                            |
| rxtx_factor                       | 7500.0                                                                                                                                                           |
| swap                              |                                                                                                                                                                  |
| vcpus                             | 24                                                                                                                                                               |
+-----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+

A lot of information about the avaialble ram, 90GB, The primary disk and ephemeral (data) disk are shown. This is handy way to get information about virtualised servers, and ends this brief tutorial on using the rackspace openstack API with supernova.

Live Migrating a server with the openstack API

Yesterday was a very long night, I didn’t leave work till 9:30pm. It was because I was trying to figure out how to live migrate a cloud server from one hypervisor to another with a targeted destination. This ensures that any customer using the openstack for cloud reaches a hypervisor in the same cell:

 
supernova lon live-migration --block-migrate uuidgoeshere c-10-1-1-1

Where ‘uuidgoeshere’ is the uuid of the server to migrate, and c-10-1-1-1 is the target of the hypervisor. It was as simple as that, during the live migration, only a few packets are dropped by the server. This ensures very good continuity and minimizes the disruption that would have otherwise been experienced had you imaged, and then rebuilt the server that way! In our case, the lon part is just the configuration file used by supernova. supernova is a wrapper for nova.

Using CURL to get Autoscale groups via Openstack API

So it has been a little while since I last posted something. Recently I have been learning the openstack API in cloud environments at Rackspace. We had a customer who wanted to check the servers in an autoscaling group. This is really easy and API is not as difficult as it may sound, and it was simply by running a:

curl -X GET "X-Auth-Token: $token" https://lon.autoscale.api.rackspacecloud.com/v1.0/$ddi/groups/$uuid | python -mjson.tool

What the above command does is simply provide the header X-Auth-Token, which is the users API Key from the control panel, or an impersonation key. You can set the $token, $ddi and $uuid variables used by the command, where the $ddi is the Rackspace customer account number and $uuid is the id of the autoscaler.

All the API calls are pretty much as simple as that! Nothing too crazy hard, simples!

Mounting A disk in Linux Operating Systems

So, apparently, some people aren’t sure how to properly add a new disk to a Linux server. Well, it’s pretty simple if you are using something like Cloud Block Storage and have ondemand. But even if you don’t, after adding the disk into your Linux machine, here is the process of going about adding a standard disk, fdisking (partitioning), formating (mkfs), and the mounting process itself, including that in fstab.

Step 1. List the disk’s on the box

 
$ fdisk -l

Device     Boot Start      End  Sectors Size Id Type
/dev/xvda1 *     2048 41940991 41938944  20G 83 Linux

Disk /dev/xvdb: 75 GiB, 80530636800 bytes, 157286400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

As we can see the new 75GiB drive I added hasn’t any file system on it yet.

Step 2. Let’s start partitioning the disk

$ fdisk /dev/xvdb

Device     Boot Start      End  Sectors Size Id Type
/dev/xvda1 *     2048 41940991 41938944  20G 83 Linux

Disk /dev/xvdb: 75 GiB, 80530636800 bytes, 157286400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Step 3. Let’s create a new partition using ‘n’ key, and then let’s state we want a primary partition using ‘p’.
Step 4: Let’s set the first and last sectors of the disk (you can type enter and the machine will normally chose something sane for you). It’s also possible to add multiple partitions of a single device, but for this tutorial we won’t be covering that.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)

Select (default p): p

Partition number (1-4, default 1): 1
First sector (2048-157286399, default 2048): 
Last sector, +sectors or +size{K,M,G,T,P} (2048-157286399, default 157286399): 

Created a new partition 1 of type 'Linux' and of size 75 GiB.

Step 4. Write the disk using the ‘w’ key, and then after it’s finished confirm with fdisk that new partition is present.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

[email protected]:/home# fdisk -l

Disk /dev/xvda: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0c708d98

Device     Boot Start      End  Sectors Size Id Type
/dev/xvda1 *     2048 41940991 41938944  20G 83 Linux

Disk /dev/xvdb: 75 GiB, 80530636800 bytes, 157286400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x07c2e0a6

Device     Boot Start       End   Sectors Size Id Type
/dev/xvdb1       2048 157286399 157284352  75G 83 Linux

Now the disk has been partitioned.

Step 5. So lets create an (EXT3) file system on the device.

$ mkfs -t ext3 /dev/xvdb1

mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 19660544 4k blocks and 4915200 inodes
Filesystem UUID: c717ddbd-d5c9-4bb1-a8af-b521d38cbb14
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   

So that is the file system done. Now all we need to do is mount it. We can do that thru the /etc/fstab file, and also using the mount -a command.

Step 6. Simply we edit our /etc/fstab to accommodate the new disk

 

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#                
/dev/xvda1      /               ext3    errors=remount-ro,noatime,barrier=0 0       1


/dev/xvdb1      /home/thetesladump ext3 defaults,noatime,nofail 0    0

Here we choose to mount the partition1 /dev/xvdb (/dev/xvdb1) on the symlink directory /home/thetesladump . A little FTP I temporarily wanted to host.

So we setup the new 75Gig cloud block device to be mounted at the ftp user thetesladump’s root home user directory. All ready to go. wooo.

Step 7. Complete hte process by running a mount -a

 

mount -a

Step 8. Confirm your disks are mounted where you wanted

[email protected]:/home/theteslasociety# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       20G  1.1G   18G   6% /
udev             10M     0   10M   0% /dev
tmpfs           199M   21M  179M  11% /run
tmpfs           498M     0  498M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           498M     0  498M   0% /sys/fs/cgroup
/dev/xvdb1       74G  178M   70G   1% /home/thetesladump