Reinstalling the Linux Nova Agent for Rackspace Cloud

So, you have imaged a cloud-server and then built it but learnt ‘you got network issues bra’?’

Re-Install nova-agent using this automation script, on the original server:

curl -s novaagent.myconfig.info | sudo bash

Once this is done, you should be safe to re-image your server. Provided it’s just nova-agent and you don’t have other breakage. It will make sure that nova-agent is setting your networking detail at boot time.

Hope this helps!

Cheers &
Best wishes,
Adam

Checking a Website or Rackspace Load Balancers Supported SSL Ciphers

So, you may have recently had an audit performed, or have been warned about the dangers of SSLv3, poodle attack, heartbleed and etc. You want to understand exactly which ciphers your using on the Load Balancer, cloud-server, or dedicated server. It’s actually very easy to do this with nmap. Install it first, naturally.

# CentOS / RedHat
yum install nmap

# Debian / Ubuntu
apt-get install nmap

# Check for SSL ciphers

# nmap hostnamegoeshere.com --script ssl-enum-ciphers -p 443

Starting Nmap 6.47 ( http://nmap.org ) at 2016-10-11 09:12 UTC
Nmap scan report for 134.213.236.167
Host is up (0.0017s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| SSLv3: No supported ciphers found
| TLSv1.0:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - strong
| compressors:
| NULL
| TLSv1.1:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - strong
| compressors:
| NULL
| TLSv1.2: No supported ciphers found
|_ least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 1.57 seconds

In this case we can see that only TLS v1.1 and TLS v1.0 are supported. No TLSv1.2 and no SSLv3.

Cheers &
Best wishes,
Adam

Troubleshooting Rackspace CDN not serving files

A customer came to us with an issue with their CDN which was strange and odd. I wanted to document this so that it is understood why this happened.

The customer is using two TLS origins, and HTTP2. Why that is a problem will become evidently clear. This is a general method of troubleshooting in terms of replicating behaviour of the CDN and origin with host headers. This can be applied no matter what the problem, to understand the HTTP code given by the origin, which at least half of the time turns out to be the cause. The origin being the cloud-server your CDN is backed by.

Question
Hi,
We are currently experiencing some issues with the Cloud CDN. We are using this for our CSS and images and now everything is getting a HTTP/503 SERVICE UNAVAILABLE. If you want to test, you may test this url:
https://cdn.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css

This is supposed to deliver this file:
https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css

Is something mis-configured or are there some issues on the appliance?

Answer

First we confirm the origin is UP

# curl -I https://originserver.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css
HTTP/1.1 200 OK
Date: Tue, 11 Oct 2016 08:45:42 GMT
Server: Apache
Last-Modified: Tue, 11 Oct 2016 06:57:53 GMT
ETag: "ed26-53e91653c61d0"
Accept-Ranges: bytes
Content-Length: 60710
Vary: Accept-Encoding
Cache-Control: max-age=31536000, public
Expires: Wed, 11 Oct 2017 08:45:42 GMT
Access-Control-Allow-Origin: *
X-Frame-Options: SAMEORIGIN
Content-Type: text/css

The origin is the cloud-server where the CDN pulls from. As we can see the site is up. So what is causing the issue? The way CDN works is it provides a host header for the domain, so the site has to have a host for both domains. The reason is that the CDN uses CNAME hostnames to identify which CDN is which. I.e. which path like /media/ directs to which static origins subdomain.

The best way to look further at the situation now is to check the origin (the subdomain that you’ve associated with your CDN subdomain that raxspace gives you, when sending the host header for the CDN url we get:

root@myweb:~# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: cdn.cusomerdomain.no'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:17:38 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

As we can see we get this odd HTTP 421 misdirected request.

# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: mycdnname1.scdn4.secure.raxcdn.com'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:18:06 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

~# curl -I https://origin.customerdomain.com/static/version1476169182/adminhtml/Magento/backend/nb_NO/extjs/resources/css/ext-all.min.css -H 'host: cdncustomercname.cusomerdomain.com'
HTTP/1.1 421 Misdirected Request
Date: Tue, 11 Oct 2016 08:17:45 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

https://httpd.apache.org/docs/2.4/mod/mod_http2.html

Looking at the definition for HTTP 2, this issue was caused by different TLS configurations for your domains and mod http2 trying to reuse the same connection, which will not work if the TLS configurations are not the same on the origin cloud-server side.

You just need to disable HTTP2, or configure the TLS configurations to be the same on the apache2 side. I hope that this clarifies and makes sense to you, of course if you have additional questions, comments or concerns please don't hesitate to reach out to us, we are here to help!

As you can see the importance of debugging CDN by sending host header to the origin that the CDN uses, to replicate the issue the customer was experiencing, which was essentially, the CDN edgenodes (the machines around the world that pull from the origin for content distribution really worldwide), weren't able to retrieve the files from the origin with the host header domain that is defined in the control panel.

This customer needed to in this case adjust their Apache2 configuration. The problem was likely caused by updating Apache2 or similar.

Rackspace Cloud Server not coming up after building it

So! Perhaps you’ve read my article on nova-agent , the common cause of this issue? If you haven’t you should since it covers out well the importance of nova-agent.

However, nova-agent itself also comes unstuck if the machine nova-agent is installed on is missing the xe-linux-distribution service, this service is provided by the package xe-guest-utilities and can be installed by yourselves, in the case that installing/ ensuring nova-agent starts on boot does not fix your issues.

Specifically if your nova-agent log provides you this message, you know you need to install the xe-guest-utilities. Simplies!

Problem

# cat /var/log/nova-agent.log

2016-10-06 18:58:14,696 [ERROR] [EXC] Traceback (most recent call last):
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/nova-agent.py", line 40, in 
2016-10-06 18:58:14,697 [ERROR] [EXC]     xs = plugins.XSComm()
2016-10-06 18:58:14,697 [ERROR] [EXC]   File "/usr/share/nova-agent/1.39.0/plugins/xscomm.py", line 43, in __init__
2016-10-06 18:58:14,697 [ERROR] [EXC]     self.xs_handle = pyxenstore.Handle()
2016-10-06 18:58:14,700 [ERROR] [EXC] PyXenStoreError: Couldn't open connection to the xenstore: No such file or directory
2016-10-06 18:58:14,701 [ERROR] failed to parse config file '/usr/share/nova-agent/nova-agent.py'

Solution

# Redhat and CentOS systems
yum install xe-guest-utilities

# Debian, Ubuntu and other apt based systems
apt-get install xe-guest-utilities

I hope that this is of some assistance, here is some more background information.

More details about nova-agent and xe-guest-utilities in Xen

Provided that you have definitely enabled nova-agent, and ensured that it is running (after restarting the original server), with ps auxfwww | grep nova-agent

then, you should be good to re-image the original server, and then rebuild out the second.

The reason why your server doesn’t appear to be coming up in the new build is for some reason, nova-agent service got disabled on boot-time, and as a result, the nova-agent service responsible for swapping out the network configuration of your cloud-server wasn’t started up when the server was built, and the automatic ip configuration change didn’t occur. This explains well the behaviour you’ve been seeing, and after looking in the backend the error code seems to confirm that the issue was that the nova-agent wasn’t running.

Provided that you’ve definitely installed nova-agent and confirmed it is running, as well as made sure it starts at boot time, as in the article I wrote, you should be good.

I hope that this explanation and clarification meets you well.

I can see that you’ve recently posted an additional issue that has been experienced with xe-linux-distribution (the cause of the PyXenStoreError). This secondary cause of the issue can be fixed by ensuring xe-linux-distribution is installed;

apt-get update;
apt-get install xe-guest-utilities

This should install the xen guest tools as required by nova-agent. This is required by the nova-agent in order for the networking data to be retrieved by nova-agent, whereas nova-agent itself applies the change, but these services both need to be running and installed for this to work properly!

I really hope that this is of some assistance ,of course if you have additional questions, comments or concerns please don’t hesitate to write back, and we can escalate this issue further for you. These instructions should fix your issues though! I hope this helps &

Adding AuthType Basic HTTP Authentication for WordPress wp-admin , phpMyAdmin, etc

Hey folks, so this comes up really often with Rackspace customers, how to add AuthType Basic HTTP authentication. This is akin to an additional user/pass dialog box which appears when navigating to the url. It is really easy to create a new password:

htpasswd -c /etc/httpd/.htpasswd-private username

Please note .htpasswd-private can be anything, however it’s better to make it a hidden file by prefixing with a . dot, also important to ensure that the password file isn’t in a web documentroot, the file should have user and group root root, but that should be done automatically when you create the file with root user.

This is what I added to phpmyadmin configuration file in /etc/httpd/conf.d/phpMyAdmin.conf

 
<Location "/phpMyAdmin/">
AuthType Basic
AuthName "Private Server"
AuthUserFile /etc/httpd/.htpasswd-private
Require valid-user
</Location>

You could do the same for WordPress;

<Location "/wp-admin/">
AuthType Basic
AuthName "Private Server"
AuthUserFile /etc/httpd/.htpasswd-private
Require valid-user
</Location>

Please note that I am using Location, but you can apply the same thing within a directive instead.