Category Archives: nova

Fixing nova-agent bugs caused by yum update

Posted on by

1. Download newest version of nova-agent from the github repo to the server you want to upgrade : https://github.com/rackerlabs/openstack-guest-agents-unix/releases

For this instance I used “nova-agent-1.39-1.x86_64.rpm” since it’s CentOS / Redhat based

2. Stop current nova-agent service

service nova-agent stop

3. Remove current in place nova-agent. I found the easiest way to do this is to just remove the entire contents of it’s directory.

rm -rf /usr/share/nova-agent/*

4. Install the new nova-agent with the RPM

rpm -ivh --nosignature nova-agent-1.39-1.x86_64.rpm

5. Start the new nova-agent service

service nova-agent start (might need to use systemd for CentOS/RHEL 7 and above)
# ie 
systemctl enable nova-agent
systemctl start nova-agent

6. Issue a networking reset to verify it is working. Check logs and verify that you see this message : ‘resetnetwork’ completed with code ‘0’, message ”

uuid=$(uuidgen)
xenstore-write data/host/$uuid '{"name":"resetnetwork","value":""}'

tail -20 /var/log/nova-agent.log

That’s it! Once you’ve done that you should reboot to verify that the nova-agent comes up on boot but otherwise nothing else is needed. Hope it helps!

Thanks to Sean from Rackspace for this.. you rock dudery.

All About NOVA and Xen Tools in Rackspace Cloud – why can’t I connect to my Windows server?

Posted on by

Why can’t I connect to my Rackspace Windows cloud-server, you ask? 2 important questions.

1. Is it a new build?
2. Is it using a custom image (a non rackspace base image).

(because the rackspace base images all have correct nova-agent and xen tools, so get networking information OK. But customer images don’t!). In the case you have run the below tests to see if nova-agent is running (or installed), you will need to install them.

Checking for the nova-agent and xe-guest-utilities 

ps auxfwww | grep nova-agent
yum -qa xe-guest-utilities nova-agent
dpkg -l xe-guest-utilities nova-agent

Explanation and solution

Thanks for reaching out to us with your inquiry today. I’m glad to convey to you that I understand what the problem is with your cloud-server not being contactable.

Main reasons for breakage

The main reason why this is not working is most likely caused by some important pieces of software being missing. There is a piece of software called nova-agent, which is responsible for setting your cloud-servers IPV4 address, network subnet/mask, and ip routes, when it is first built. This is important, since the server image you built the server from, has different network details.

The rackspace build process giving networking detail to the VM is completely dependent on xe-guest-utilities and nova-agent

What has happened in this case, because the nova-agent wasn’t running on the cloud-server, the hypervisor software Rackspace use to automate cloud-server builds wasn’t able to contact the nova-agent running on your cloud-server, and therefore nova-agent wasn’t able to update the networking information. And hence, your not able to connect to it on it’s IPv4 address you are given at build time.

The steps to resolution: installing nova-agent and xen guest utilities
As such, nova-agent needs to be installed on the cloud-server you take the image from, it can be installed as follows:

https://community.rackspace.com/products/f/25/t/5694

Also nova-agent uses another piece of important software called xe-guest-utilities, or (Xen Tools) for your windows servers, this is an important ‘PV’ paravirtualization tools, responsible for seamless management of cloud-servers. Sorry that in this case it’s not working out seamlessly, but this can happen with images taken of servers which have had nova-agent disabled, uninstalled, or similar.

Upgrading the tools that nova-agent depends upon, can be installed by following the instructions at the following location:

https://support.rackspace.com/how-to/upgrade-citrix-xen-server-tools-for-windows-cloud-servers/

# Options of how to do this / Summary of Solution Steps

Naturally, you might be wondering how to achieve these changes, if you cannot RDP to the server. This is quite understandable, there are two ways to get this working;

Option 1) Manually install nova-agent on the current server you cannot access, then manually install the Xen Tools in the same way. This shall fix the OS on the server itself, and not the original image you built the server from. So it is important to create a new cloud-server image after performing these steps and us verifying tools + nova-agent installed correctly.

2) Manually install nova-agent on the source server you initially taken the image from, and install Xen Tools, then re-image the server, and then re-deploy. This should seamlesssly work each time on build with that image, provided the tools are installed. You will not need to recreate the image, since your fixing the problem on the cloud-server source that the original image was taken from.

I appreciate that these things are not 100% simple to get your head around and can be confusing for customers, I hope my explanation and summary makes this a little more painless to fix. Of course if you have additional questions, comments or concerns or don’t understand something I’ve said, please don’t hesitate to reach out to us, we are here to help!

Using Nova/Supernova to increase quota allotments in Openstack

Posted on by

I noticed there were some changes to the way we used openstack quotas today. So I had to do it the manual way! Please note that this can only be done thru the Admin API function, so if you are a Rackspace customer you would need to reach out to us to do this, unless you ran your own openstack or devstack implementation in-house.

Here is how I achieved it:

supernova {environment} {action} --option {number} {customer_ID}

supernova lon quota-update --instances 250 10010101

There is a lot of different commands available, use nova help to get more detail

 supernova lon help quota-update
[SUPERNOVA] Running nova against lon...
usage: nova quota-update [--user ] [--instances ]
                         [--cores ] [--ram ]
                         [--floating-ips ]
                         [--fixed-ips ]
                         [--metadata-items ]
                         [--injected-files ]
                         [--injected-file-content-bytes ]
                         [--injected-file-path-bytes ]
                         [--key-pairs ]
                         [--security-groups ]
                         [--security-group-rules ]
                         [--server-groups ]
                         [--server-group-members ]
                         [--force]
                         

Update the quotas for a tenant/user.

Positional arguments:
                     ID of tenant to set the quotas for.

Optional arguments:
  --user               ID of user to set the quotas for.
  --instances        New value for the "instances" quota.
  --cores                New value for the "cores" quota.
  --ram                    New value for the "ram" quota.
  --floating-ips 
                                New value for the "floating-ips" quota.
  --fixed-ips        New value for the "fixed-ips" quota.
  --metadata-items 
                                New value for the "metadata-items" quota.
  --injected-files 
                                New value for the "injected-files" quota.
  --injected-file-content-bytes 
                                New value for the "injected-file-content-
                                bytes" quota.
  --injected-file-path-bytes 
                                New value for the "injected-file-path-bytes"
                                quota.
  --key-pairs        New value for the "key-pairs" quota.
  --security-groups 
                                New value for the "security-groups" quota.
  --security-group-rules 
                                New value for the "security-group-rules"
                                quota.
  --server-groups 
                                New value for the "server-groups" quota.
  --server-group-members 
                                New value for the "server-group-members"
                                quota.
  --force                       Whether force update the quota even if the
                                already used and reserved exceeds the new
                                quota.

Resetting Rax_Service_Level_Automation metadata

Posted on by

This one is worth a mention because it causes some of our customers alarm. If your seeing this ‘warning’ in your Cloud-server control panel, don’t fret!

Building” “Server build complete. Installing & configuring software.

It’s just automation!

When you build a cloud-server and select the two tick boxes at the bottom of the build server page (scroll right down), this instructs rackspace automation to attempt to install the Rackspace monitoring & Rackspace Backup agent.

When the server finishes building, these are usually applied by the automation, but sometimes it may have an issue logging into the server and doesn’t run as expected.

Since this warning only indicates the monitoring and cloud backup auto-install failed, these can still be installed by yourselves manually at the below location (please note these links may become out of date use docs.rackspace.com and support.rackspace.com for more detail):

# Manually Install Monitoring (if required)
https://support.rackspace.com/how-to/install-and-configure-the-rackspace-monitoring-agent/

# Manually Install Cloud Backup (if required)
https://support.rackspace.com/how-to/rackspace-cloud-backup-install-the-agent-on-linux/

To summarise and clarify, the notification ‘”Server build complete. Installing & configuring software.’ indicates that your server environment built OK, and that the server is waiting for automation to install the additional 2 Rackspace products.

If you see this notification again, it is safe to ignore in terms of the functioning of the cloud-server, and is intended as a warning so you know Rackspace monitoring and cloud backup were not additionally installed by the automation. I have reset the state of your server and you can consider this situation resolved.

If this is causing you concern, it’s actually possible to correct this yourself by installing supernova and novaclient. Please take special care when using admin resources such as nova and API. It’d be difficult to break something if you don’t follow these instructions, but still…take care!

# Install using Python pip the supernova nova wrapper and the rackspace-novaclient
pip install supernova rackspace-novaclient
# Remove the 'rax_service_level_automation' metadata from this server
supernova customer meta serveruuidgoeshere delete rax_service_level_automation

Simples fix. Please note that you will need to configure supernova. This is explained in the supernova category of this blog, and also at:

 

https://developer.rackspace.com/blog/supernova-managing-openstack-environments-made-easy/
https://github.com/major/supernova
https://media.readthedocs.org/pdf/supernova/latest/supernova.pdf

Cheers &

Best wishes,
Adam

Windows Password reset for Rackspace Cloud Servers

Posted on by

In the previous articles Using API and BASH to validate changing conditions and Reset windows administrator password using rescue mode without nova-agent I explained both the steps how to reset the password of a windows VM instance by modifying the SAM file by using a Linux ‘rescue’ image in the cloud, and, I also explained how to automate checks for BASH automation thru the API. The checks specifically waited until the server entered rescue, and then lifted the ipv4 address, connecting only when the rescue server had finished building.

That way the automation is handling the delay it takes, as well as setting and lifting the access credentials and ip address to use each time. Here is the complete script. Please note that backticks are deprecated but I’m a bit ‘oldskool’. This is a rough alpha, but it works really nicely. After testing it consistently allows ourselves, or our customers to reset a Windows Cloud Server password, in the case that a customer loses access to it, and cannot use other Rackspace services to do the reset. This effectively turns a useless server, back into a usable one again and saves a lot of time.

#!/bin/bash
# Adam Bull, Rackspace UK
# This script automates the resetting of windows passwords
# Arguments $1 == username
# Arguments $2 == apikey
# Arguments $3 == ddi
# Arguments $4 == instanceid

echo "Rackspace windows cloud server Password Reset"
echo "written by Adam Bull, Rackspace UK"
sleep 2
PASSWORD=39fdfgk4d3fdovszc932456j2oZ

# Provide an instance uuid to rescue and reset windows password

USERNAME=mycloudusernamehere
APIKEY=myapikeyhere
# DDI is the 'customer ID', if you don't know this login to the control panel and check the number in the URL
DDI=10010101
# The instance uuid you want to rescue
INSTANCE=ca371a8b-748e-46da-9e6d-8c594691f71c

# INITIATE RESCUE PROCESS

nova  --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure rescue --password "$PASSWORD" --image 7fade26a-0cca-415f-a988-49c021768fca $INSTANCE

# LOOP UNTIL STATE DETECTED AS RESCUED

STATE=0
until [[ $STATE == rescued ]]; do
echo "start rescue check"
STATE=`nova --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure show $INSTANCE | grep rescued | awk '{print $4}'`

echo "STATE =" $STATE
echo "sleeping.."
sleep 5
done

# EXTRACT PUBLIC ipv4 FROM INSTANCE

IP=`nova --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure show $INSTANCE | grep public | awk '{print $5}' | grep -oE '((1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])\.){3}(1?[0-9][0-9]?|2[0-4][0-9]|25[0-5])'`
echo "IP = $IP"

# UPDATE AND INSTALL RESCUE TOOLS AND RESET WINDOWS PASS
# Set environment locally
yum install sshpass -y

# Execute environment remotely
echo "Performing Rescue..."
sshpass -p "$PASSWORD" ssh -o StrictHostKeyChecking=no root@"$IP" 'yum update -y; yum install ntfs-3g -y; mount /dev/xvdb1 /mnt; curl li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm -o /root/nux.rpm; rpm -Uvh /root/nux.rpm; yum install chntpw -y; cd /mnt/Windows/System32/config; echo -e "1\ny\n" | chntpw -u "Administrator" SAM'

echo "Unrescuing in 100 seconds..."
sleep 100
nova  --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure unrescue $INSTANCE

Thanks again to my friend Cory who gave me the instructions, I simply automated the process to make it easier and learned something in the process 😉

HOWTO: Rackspace Automation, Using BASH with API (to validate conditions to perform conditional tasks)

Posted on by

In the previous article, I showed how to wipe clean the windows password from a broken Virtual Machine that you were locked out of by rescuing with a Linux image. In this article I explain steps of how you would automate this with a bash script, that looked at the STATE of the server, and accepts commandline arguments.

It’s quite a simple script;

#!/bin/bash
# Adam Bull
# April 28 2016
# This script automates the resetting of windows passwords
# Arguments $1 == instanceuuid
# Arguments $2 == username
# Arguments $3 == apikey

PASSWORD=mypassword

# Provide an instance uuid to rescue and reset windows password

USERNAME=$1
APIKEY=$2
DDI=$3
INSTANCE=$4


nova  --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure rescue --password mypassword --image 7fade26a-0cca-415f-a988-49c021768fca $INSTANCE

The above script takes the arguments I give the executable script on the commandline, in this case the first argument passed is $1, the Rackspace mycloud username. The second argument the apikey. etc. This basically puts the server into rescue. But.. what if we wanted to run some automation AFTER it rescued? We don’t want to try and let the automation ssh to the box and run the automation early, so we could use a supernova show to find whether the VM state has changed to ‘rescue’. Whilst its initiating the state will be rescuing. So we have the option of using when !rescueing logic, or, when == equal to rescue. Lets use when equal to rescue in our validation loop.

This loop will continue until the task state changes to the desired value. Here is how we achieve it

#!/bin/bash
# Initialize Variable
STATE=0
# Validate $STATE variable, looping UNTIL $STATE == rescued
until [[ $STATE == rescued ]]; do
echo "start rescue check"
# 'show' the servers data, and grep for rescued and extract only the correct value if it is found
STATE=`nova --os-username $USERNAME --os-auth-system=rackspace  --os-tenant-name $DDI --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password $APIKEY --insecure show $INSTANCE | grep rescued | awk '{print $4}'`

# For debugging
echo "STATE =" $STATE
echo "sleeping.."

# For API Limit control
sleep 5
# Exit the loop once until condition satisfied
done

# Post Rescue
echo "If you read this, it means that the program detected a rescued state"

It’s quite a simple script to use. We just provide the arguments $1, $2, $3 and $4. 

 
./rescue.sh mycloudusername mycloudapikey 10010101 e744af0f-6643-44f4-a63f-d99db1588c94

Where 10010101 is the tenant id and e744af0f-6643-44f4-a63f-d99db1588c94 is the UUID of your server.

It’s really quite simple to do! But this is not enough we want to go a step further. Let’s move the rescue.sh to /bin

# WARNING /bin is not a playground this is for demonstration purposes of how to 'install' bin system applications
cp rescue.sh /bin/rescue.sh

Now you can call the command ‘rescue’.

rescue mycloudusername mycloudapikey mycustomerid mycloudserveruuidgoeshere

nice, and quite simple too. Obviously ‘post rescue’ in the script I can upload a script via ssh to the server, and then execute it remotely to perform the password reset.

Reset Windows Administrator Password on Rackspace Cloud Server using Rescue Mode (without nova-agent)

Posted on by

So, you have lost your Windows Administrator password for your Rackspace cloud server? I’d like to thank my friend Cory for providing the link details for how to do this.

No problem. Simply put the Windows VM into rescue mode using a Linux image (yup!)

Put Windows VM into Rescue mode using Linux image 

# Initiate rescue using the CentOS 7 image for the server uuid 0b67faf7-bc56-4844-ad0b-16e39f289ef6
$ nova me rescue --password mypasswordforrescuemodehere --image 7fade26a-0cca-415f-a988-49c021768fca 0b67faf7-bc56-4844-ad0b-16e39f289ef6

If you’ve broken your Rackspace server and you don’t know how to perform the above step, send a ticket to Rackspace support and they should be able to put your server in rescue so you can reset the password of your windows machine!

SSH to rescue server 

ssh root@myserveriphere

Check which disk is Windows NTFS 

# fdisk -l

Disk /dev/xvdc: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x0003e9b3

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdc1            2048     4194303     2096128   83  Linux

Disk /dev/xvdb: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xfcb073fc

    Device Boot      Start         End      Blocks   Id  System
/dev/xvdb1   *        2048   167770111    83884032    7  HPFS/NTFS/exFAT

Disk /dev/xvda: 85.9 GB, 85899345920 bytes, 167772160 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00070dc0

Here we can see that the disk we want is /dev/xvdb1 since this is the HPFS/NTFS/exFAT partition format used by windows. The rescue mode builds a new server and disk, attaching your old disk as the ‘b’ disk, xvdb. Lets mount the disk and install the application we need to wipe the password for the box.

Mount the disk 

yum update -y
yum install ntfs-3g -y
mount /dev/xvdb1 /mnt

Download and install the chntpw tool 

 
curl li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm -o /root/nux.rpm
rpm -Uvh /root/nux.rpm
yum install chntpw -y

Run tool against Windows Administrator SAM file

Once run press 1 to ‘clear password’. 

root@RESCUE-test config]# chntpw -u "Administrator" SAM
chntpw version 0.99.6 110511 , (c) Petter N Hagen
Hive  name (from header): <\SystemRoot\System32\Config\SAM>
ROOT KEY at offset: 0x001020 * Subkey indexing type is: 666c 
File size 262144 [40000] bytes, containing 6 pages (+ 1 headerpage)
Used for data: 255/20712 blocks/bytes, unused: 13/3672 blocks/bytes.


* SAM policy limits:
Failed logins before lockout is: 0
Minimum password length        : 0
Password history count         : 0
| RID -|---------- Username ------------| Admin? |- Lock? --|
| 01f4 | Administrator                  | ADMIN  |          |
| 01f5 | Guest                          |        | dis/lock |

---------------------> SYSKEY CHECK <-----------------------
SYSTEM   SecureBoot            : -1 -> Not Set (not installed, good!)
SAM      Account\F             : 0 -> off
SECURITY PolSecretEncryptionKey: -1 -> Not Set (OK if this is NT4)
Syskey not installed!

RID     : 0500 [01f4]
Username: Administrator
fullname:
comment : Built-in account for administering the computer/domain
homedir :

User is member of 1 groups:
00000220 = Administrators (which has 1 members)

Account bits: 0x0010 =
[ ] Disabled        | [ ] Homedir req.    | [ ] Passwd not req. |
[ ] Temp. duplicate | [X] Normal account  | [ ] NMS account     |
[ ] Domain trust ac | [ ] Wks trust act.  | [ ] Srv trust act   |
[ ] Pwd don't expir | [ ] Auto lockout    | [ ] (unknown 0x08)  |
[ ] (unknown 0x10)  | [ ] (unknown 0x20)  | [ ] (unknown 0x40)  |

Failed login count: 0, while max tries is: 0
Total  login count: 15

- - - - User Edit Menu:
 1 - Clear (blank) user password
 2 - Edit (set new) user password (careful with this on XP or Vista)
 3 - Promote user (make user an administrator)
(4 - Unlock and enable user account) [seems unlocked already]
 q - Quit editing user, back to user select
Select: [q] > 1
Password cleared!

Hives that have changed:
 #  Name
 0  
Write hive files? (y/n) [n] : y
 0   - OK

It’s been done, yay!

Unrescue the cloud server, either from control panel or using nova 

abull-mb:~ adam$ supernova me unrescue 0b67faf7-bc56-4844-ad0b-16e39f289ef6

Yay! We now automatically bypass the ordinary login screen so we can get into the server to reconfigure it properly again.
Screen Shot 2016-04-29 at 11.30.43 AM

You might have some questions about… setting up nova.

Setting up Nova 

# Nova configuration

#export OS_AUTH_URL=https://lon.identity.api.rackspacecloud.com/v2.0/
#export OS_AUTH_SYSTEM=rackspace_uk
#export OS_REGION_NAME=LON
#export OS_USERNAME=mycloudusernamehere
# Tenant Name is customer number shown in url of mycloud control panel
##export OS_TENANT_NAME=10101010
#export NOVA_RAX_AUTH=1
#export OS_PASSWORD=mycloudapikeyhere
# Project ID is customer number shown in url of mycloud control panel
#export OS_PROJECT_ID=100101010
#export OS_NO_CACHE=1

These ‘environment variables’ should be put in a file like your .bash_profile. Then you will want to source it before using nova

source .bash_profile
or
. .bash_profile

This just sets the variables on the commandline so they can be used by nova. It is possible to provide all of the credentials on the nova commandline as described in previous articles on this blog concerning nova.

Using nova without .bash_profile or environment variables

Initiate Rescue Mode

nova --os-username mycloudusernamegoeshere --os-auth-system=rackspace  --os-tenant-name tenantidgoeshere --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password apigoeshere rescue --password mypasswordforrescuemodehere --image 7fade26a-0cca-415f-a988-49c021768fca 0b67faf7-bc56-4844-ad0b-16e39f289ef6

Un-rescue 

nova --os-username mycloudusernamegoeshere --os-auth-system=rackspace  --os-tenant-name tenantidgoeshere --os-auth-url https://lon.identity.api.rackspacecloud.com/v2.0/ --os-password apigoeshere unrescue

Installing nova

for more details about how to install python based nova, used in this article, please see;
https://support.rackspace.com/how-to/installing-python-novaclient-on-linux-and-mac-os/

Creating 200 cloud servers using openstack Nova

Posted on by

Had a question on how to do this from a customer today.
It is possible to create very many cloud servers in a quick time something like:

#!/bin/sh
for i in `seq 1 200`;
do
nova boot --image someimageidhere --flavor '2GB Standard Instance' "\Server-$i"
sleep 5
done

So simple, but could build out many servers (a small farm) in just an hour or so:D

Update

So my colleague tells me, that backticks are bad, i.e. deprecated. Which, they are, and I expected to hear this from someone, as my knowledge is somewhat a little old school. Here is what my friend recommends.

for i in {0..200}; do
nova boot --image someimageidhere --flavor '2GB Standard Instance' "\Server-$i"
sleep 5
done

Resizing a Rackspace Performance Server

Posted on by

It’s possible for the customer to do this thru the API, but it is without express warantee. It’s not possible to resize performance servers thru the mycloud control panel, so, to do it you will need to use curl API, or what I like to use, supernova wrapper for nova or nova. It’s quite simple really;

The below example is how to resize a performance server to 4 gigs (this was from 2 gigs)

supernova customer resize --poll uuidgoeshere performance1-4