Enabling MySQL Slow Query Logs

In the case your seeing very long pageload times, and have checked your application. It always pays to check the way in which the database performs when interacting with the application, especially if they are on either same or seperate server, as these significantly affect the way that your application will run.


mysql> SET GLOBAL slow_query_log = 'ON' ;
Query OK, 0 rows affected (0.01 sec)

mysql> SET GLOBAL slow_query_log_file = '/slow_query_logs/slow_query_logs.txt';
Query OK, 0 rows affected (0.00 sec)

mysql> SET GLOBAL long_query_time = 5;
Query OK, 0 rows affected (0.00 sec)

Checking Webserver Logs and generating hit counts

So, I’ve been meaning to do this for a while. I’m sure your all familiar with this crappy oneliner where you’ll check the hits using wc -l against a webserver log for a given hour and use for i in seq or similar to get the last 10 minutes data, or the last 59 minutes. But getting hours and days is a bit harder. You need some additional nesting, and, it isn’t difficult for you to do at all!

for i in 0{1..9}; do echo "16/Jan/2017:06:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/soemsite.com-access_log | wc -l ; done

I improved this one, quite significantly by adding an additional for j, for the hour, and adding an additional 0 in front of {1..9}, this properly is matching the Apache2 log format and allows me to increment through all the hours of the day. to present ;-D All that is missing is some error checking when the last date in the file is, im thinking a tail and grep for the timecode from the log should be sufficient.

Here is the proud oneliner I made for this!

for j in 0{1..9}; do for i in 0{1..9} {10..59}; do echo "16/Jan/2017:$j:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/website.com-access_log | wc -l ; done; done

Upgrading PHP 5.3.29 to PHP 7 on Centos 6.8 Using Rackspace IUS Repo

These instructions only apply in specific cases. Specifically CentOS machines, running in the Rackspace Cloud, IUS the Rackspace provided repo, provides several things not usually available within the CentOS repo, without you manually compiling more recent versions. One of them is the latest version of PHP7.0 and PHP7.1.

I wanted to quickly document the process, since it is a relatively simple process, and, can actually be done without any maintenance window, if you know what your doing, with very minimal, (if any) disruption to running sites. an apachectl graceful, actually, should be enough. Since apachectl gracefully restarts apache httpd, the downtime you’ll see will be super minimal. Expect nobody to notice you upgraded to PHP7 if you do this right.

If you do this incorrectly, you will break the PHP installation, and worse, break all of the sites using mod_php. Lets take a look at the steps:

Step 1. Check available PHP modules provided by presently configured REPO

root@server3 ~]# yum search php7
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
============================================================================================================================= N/S Matched: php7 ==============================================================================================================================
php70u-debuginfo.x86_64 : Debug information for package php70u
php70u-ioncube-loader-debuginfo.x86_64 : Debug information for package php70u-ioncube-loader
php70u-pecl-amqp-debuginfo.x86_64 : Debug information for package php70u-pecl-amqp
php70u-pecl-apcu-debuginfo.x86_64 : Debug information for package php70u-pecl-apcu
php70u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php70u-pecl-igbinary
php70u-pecl-imagick-debuginfo.x86_64 : Debug information for package php70u-pecl-imagick
php70u-pecl-redis-debuginfo.x86_64 : Debug information for package php70u-pecl-redis
php70u-pecl-smbclient-debuginfo.x86_64 : Debug information for package php70u-pecl-smbclient
php70u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php70u-pecl-xdebug
php71u-debuginfo.x86_64 : Debug information for package php71u
php71u-pecl-apcu-debuginfo.x86_64 : Debug information for package php71u-pecl-apcu
php71u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php71u-pecl-igbinary
php71u-pecl-redis-debuginfo.x86_64 : Debug information for package php71u-pecl-redis
php71u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php71u-pecl-xdebug
sclo-php70-php-pecl-propro-devel.x86_64 : sclo-php70-php-pecl-propro developer files (header)
sclo-php70-php-pecl-raphf-devel.x86_64 : sclo-php70-php-pecl-raphf developer files (header)
uwsgi-plugin-php70u-debuginfo.x86_64 : Debug information for package uwsgi-plugin-php70u
mod_php70u.x86_64 : PHP module for the Apache HTTP Server
mod_php71u.x86_64 : PHP module for the Apache HTTP Server
php70u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php70u-cli.x86_64 : Command-line interface for PHP
php70u-common.x86_64 : Common files for PHP
php70u-dba.x86_64 : A database abstraction layer module for PHP applications
php70u-dbg.x86_64 : The interactive PHP debugger
php70u-devel.x86_64 : Files needed for building PHP extensions
php70u-embedded.x86_64 : PHP library for embedding in applications
php70u-enchant.x86_64 : Enchant spelling extension for PHP applications
php70u-fpm.x86_64 : PHP FastCGI Process Manager
php70u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php70u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php70u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php70u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php70u-imap.x86_64 : A module for PHP applications that use IMAP
php70u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php70u-intl.x86_64 : Internationalization extension for PHP applications
php70u-ioncube-loader.x86_64 : IonCube Loader provides PHP Modules to read IonCube Encoded Files
php70u-json.x86_64 : JavaScript Object Notation extension for PHP
php70u-ldap.x86_64 : A module for PHP applications that use LDAP
php70u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php70u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php70u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php70u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php70u-opcache.x86_64 : The Zend OPcache
php70u-pdo.x86_64 : A database access abstraction module for PHP applications
php70u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php70u-pear.noarch : PHP Extension and Application Repository framework
php70u-pecl-amqp.x86_64 : Communicate with any AMQP compliant server
php70u-pecl-apcu.x86_64 : APC User Cache
php70u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php70u-pecl-apcu-panel.noarch : APCu control panel
php70u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php70u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php70u-pecl-imagick.x86_64 : Provides a wrapper to the ImageMagick library
php70u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php70u-pecl-smbclient.x86_64 : PHP wrapper for libsmbclient
php70u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php70u-pgsql.x86_64 : A PostgreSQL database module for PHP
php70u-process.x86_64 : Modules for PHP script using system process interfaces
php70u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php70u-recode.x86_64 : A module for PHP applications for using the recode library
php70u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php70u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php70u-tidy.x86_64 : Standard PHP module provides tidy library support
php70u-xml.x86_64 : A module for PHP applications which use XML
php70u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
php71u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php71u-cli.x86_64 : Command-line interface for PHP
php71u-common.x86_64 : Common files for PHP
php71u-dba.x86_64 : A database abstraction layer module for PHP applications
php71u-dbg.x86_64 : The interactive PHP debugger
php71u-devel.x86_64 : Files needed for building PHP extensions
php71u-embedded.x86_64 : PHP library for embedding in applications
php71u-enchant.x86_64 : Enchant spelling extension for PHP applications
php71u-fpm.x86_64 : PHP FastCGI Process Manager
php71u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php71u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php71u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php71u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php71u-imap.x86_64 : A module for PHP applications that use IMAP
php71u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php71u-intl.x86_64 : Internationalization extension for PHP applications
php71u-json.x86_64 : JavaScript Object Notation extension for PHP
php71u-ldap.x86_64 : A module for PHP applications that use LDAP
php71u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php71u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php71u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php71u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php71u-opcache.x86_64 : The Zend OPcache
php71u-pdo.x86_64 : A database access abstraction module for PHP applications
php71u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php71u-pecl-apcu.x86_64 : APC User Cache
php71u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php71u-pecl-apcu-panel.noarch : APCu control panel
php71u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php71u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php71u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php71u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php71u-pgsql.x86_64 : A PostgreSQL database module for PHP
php71u-process.x86_64 : Modules for PHP script using system process interfaces
php71u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php71u-recode.x86_64 : A module for PHP applications for using the recode library
php71u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php71u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php71u-tidy.x86_64 : Standard PHP module provides tidy library support
php71u-xml.x86_64 : A module for PHP applications which use XML
php71u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
sclo-php70-php-pecl-apcu.x86_64 : APC User Cache
sclo-php70-php-pecl-apcu-bc.x86_64 : APCu Backwards Compatibility Module
sclo-php70-php-pecl-apcu-devel.x86_64 : APCu developer files (header)
sclo-php70-php-pecl-apfd.x86_64 : Always Populate Form Data
sclo-php70-php-pecl-http.x86_64 : Extended HTTP support
sclo-php70-php-pecl-http-devel.x86_64 : Extended HTTP support developer files (header)
sclo-php70-php-pecl-lzf.x86_64 : Extension to handle LZF de/compression
sclo-php70-php-pecl-mongodb.x86_64 : MongoDB driver for PHP
sclo-php70-php-pecl-propro.x86_64 : Property proxy
sclo-php70-php-pecl-raphf.x86_64 : Resource and persistent handles factory
sclo-php70-php-pecl-selinux.x86_64 : SELinux binding for PHP scripting language
sclo-php70-php-pecl-solr2.x86_64 : Object oriented API to Apache Solr
sclo-php70-php-pecl-uploadprogress.x86_64 : An extension to track progress of a file upload
sclo-php70-php-pecl-uuid.x86_64 : Universally Unique Identifier extension for PHP
sclo-php70-php-pecl-xattr.x86_64 : Extended attributes
sclo-php70-php-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
uwsgi-plugin-php70u.x86_64 : uWSGI - Plugin for PHP support

  Name and summary matches only, use "search all" for everything.

As we can see php7 is there. Great. But what about the php packages they have already? That’s coming up next.

Step 2. Check presence of plugin replace, we’ll use this to upgrade to mod_php70 once we’re ready

# Locate plugin replace is available
yum search yum-plugin-replace
# Install yum plugin replace if available (otherwise this will not work for you as easily)
yum install yum-plugin-replace

Step 3. Run a mock yum replace

# yum replace php53u --replace-with mod_php70u.x86_64
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
Error: No Package Matching mod_php70u.x86_64
[root@server3 ~]# yum replace php53u --replace-with mod_php70u
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00

WARNING: Unable to resolve all providers: ['config(php53u-common)', 'curl.so()(64bit)', 'fileinfo.so()(64bit)', 'json.so()(64bit)', 'phar.so()(64bit)', 'php-api', 'php-pecl(Fileinfo)', 'php-pecl(phar)', 'php-pecl(zip)', 'php-pecl-Fileinfo', 'php-pecl-phar', 'php-pecl-zip', 'php-zend-abi', 'php53(api)', 'php53(language)', 'php53(zend-abi)', 'php53-api', 'php53-bz2', 'php53-calendar', 'php53-common', 'php53-ctype', 'php53-curl', 'php53-date', 'php53-exif', 'php53-filter', 'php53-ftp', 'php53-gettext', 'php53-gmp', 'php53-hash', 'php53-iconv', 'php53-json', 'php53-libxml', 'php53-openssl', 'php53-pcre', 'php53-pecl(Fileinfo)', 'php53-pecl(json)', 'php53-pecl(phar)', 'php53-pecl(zip)', 'php53-pecl-Fileinfo', 'php53-pecl-json', 'php53-pecl-phar', 'php53-pecl-zip', 'php53-posix', 'php53-reflection', 'php53-session', 'php53-shmop', 'php53-simplexml', 'php53-sockets', 'php53-spl', 'php53-sqlite3', 'php53-sysvmsg', 'php53-sysvsem', 'php53-sysvshm', 'php53-tokenizer', 'php53-wddx', 'php53-zend-abi', 'php53-zip', 'php53-zlib', 'php53u(api)', 'php53u(language)', 'php53u(zend-abi)', 'php53u-api', 'php53u-bz2', 'php53u-calendar', 'php53u-ctype', 'php53u-curl', 'php53u-date', 'php53u-exif', 'php53u-fileinfo', 'php53u-filter', 'php53u-ftp', 'php53u-gettext', 'php53u-gmp', 'php53u-hash', 'php53u-iconv', 'php53u-json', 'php53u-libxml', 'php53u-openssl', 'php53u-pcre', 'php53u-pecl(Fileinfo)', 'php53u-pecl(json)', 'php53u-pecl(phar)', 'php53u-pecl(zip)', 'php53u-pecl-Fileinfo', 'php53u-pecl-json', 'php53u-pecl-phar', 'php53u-pecl-zip', 'php53u-posix', 'php53u-reflection', 'php53u-session', 'php53u-shmop', 'php53u-simplexml', 'php53u-sockets', 'php53u-spl', 'php53u-sqlite3', 'php53u-sysvmsg', 'php53u-sysvsem', 'php53u-sysvshm', 'php53u-tokenizer', 'php53u-wddx', 'php53u-zend-abi', 'php53u-zip', 'php53u-zlib', 'zip.so()(64bit)', 'php53u-common', 'php53u-common(x86-64)', 'php53-cgi', 'php53-cli', 'php53-pcntl', 'php53-readline', 'php53u-cgi', 'php53u-pcntl', 'php53u-readline', 'php53u-cli', 'php53u-cli(x86-64)', 'config(php53u-pdo)', 'pdo.so()(64bit)', 'pdo_sqlite.so()(64bit)', 'php53-pdo', 'php53-pdo-abi', 'php53-pdo_sqlite', 'php53u-pdo-abi', 'php53u-pdo', 'php53u-pdo(x86-64)', 'config(php53u-mysql)', 'mysql.so()(64bit)', 'mysqli.so()(64bit)', 'pdo_mysql.so()(64bit)', 'php-mysql', 'php53-mysql', 'php53-mysqli', 'php53u-mysqli', 'php53u-mysql', 'php53u-mysql(x86-64)', 'config(php53u)', 'libphp5.so()(64bit)', 'mod_php53u', 'php53', 'php53u', 'php53u(x86-64)', 'libphp5.so()(64bit)', 'php53-zts', 'php53u-zts', 'php53u-zts(x86-64)']

This may be normal depending on the package.  Continue? [y/N] y
Resolving Dependencies
--> Running transaction check
---> Package mod_php70u.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php53u.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-cli.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-common.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-mysql.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pdo.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pear.noarch 1:1.9.4-3.ius.centos6 will be erased
---> Package php53u-zts.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php70u-cli.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-common.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-gmp.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-json.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-mysqlnd.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pdo.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pear.noarch 1:1.10.1-2.ius.centos6 will be installed
---> Package php70u-process.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-xml.x86_64 0:7.0.14-3.ius.centos6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                        Version                                                                      Repository                                                 Size
==============================================================================================================================================================================================================================================================================
Installing:
 mod_php70u                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       2.7 M
 php70u-cli                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       4.0 M
 php70u-common                                                       x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       1.1 M
 php70u-gmp                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        65 k
 php70u-json                                                         x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        62 k
 php70u-mysqlnd                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       221 k
 php70u-pdo                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       115 k
 php70u-pear                                                         noarch                                                      1:1.10.1-2.ius.centos6                                                       ius                                                       362 k
 php70u-process                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        72 k
 php70u-xml                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       183 k
Removing:
 php53u                                                              x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.4 M
 php53u-cli                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      7.9 M
 php53u-common                                                       x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      3.4 M
 php53u-mysql                                                        x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      219 k
 php53u-pdo                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      126 k
 php53u-pear                                                         noarch                                                      1:1.9.4-3.ius.centos6                                                        @ius                                                      2.2 M
 php53u-zts                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.6 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install      10 Package(s)
Remove        7 Package(s)

Total download size: 8.8 M
Is this ok [y/N]: N
Exiting on user Command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx-2017-01-13-10-57L3T7JK.yumtx
You have mail in /var/spool/mail/root

Naturally, if you are satisfied that you do not need php53u-zts, the only php module which is not supported by PHP7, then you can proceed.

If you are wondering what ZTS is, The php-zts package contains a module for use with the Apache HTTP
Server which can operate under a threaded server processing model. (source pbone.net CentOS REPO)

ZTS is not required for MPM prefork, and is generally only used with MPM worker, afaik. So as long as your using prefork apache httpd your fine;

# apachectl -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

In our case prefork is being used, not worker. So I don’t think ZTS being missing is going to affect us. So we can proceed with typing ‘y’.

And’ thats pretty much how you upgrade to php7, it’s really easy with Rackspace IUS.

Troubleshooting Akamai CDN using Pragma Headers

Rackspace Cloud Files CDN enabled containers and the Rackspace CDN product can occasionally have an issue, in such cases, you can troubleshoot this a lot easier by using the DEBUG headers. To do this add -D and the folowing -H headers for your output

curl -I http://rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com/common/test/generator.png -D - -H "Pragma: akamai-x-get-client-ip, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-check-cacheable, akamai-x-get-cache-key, akamai-x-get-extracted-values, akamai-x-get-nonces, akamai-x-get-ssl-client-session-id, akamai-x-get-true-cache-key, akamai-x-serial-no, akamai-x-feo-trace, akamai-x-get-request-id" -L

Naturally for this to work, you need to make sure you have a test URL. You can get one from Cloud Files in the Rackspace Control panel, or from whatever origin is configured with the CDN. Just login to the origin, and find a file path like httpdocs/somefolder/somefile.png and then get the raxcdn.com URL from the Rackspace CDN product page, or the Rackspace Cloud Files ‘show all links’ page on the cog icon next to the cloud files container. And add the path behind your documentroot.

So for the CDN URL rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com

For the files on the origin in the documentroot of the website configured with the CDN just add the ‘local’ document path within httpdocs or your www folder!

i.e. rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com becomes rackspacecdnurlgoeshereprivatecensored.r17.cf3.rackcdn.com/somefolder/somefile.png

All About NOVA and Xen Tools in Rackspace Cloud – why can’t I connect to my Windows server?

Why can’t I connect to my Rackspace Windows cloud-server, you ask? 2 important questions.

1. Is it a new build?
2. Is it using a custom image (a non rackspace base image).

(because the rackspace base images all have correct nova-agent and xen tools, so get networking information OK. But customer images don’t!). In the case you have run the below tests to see if nova-agent is running (or installed), you will need to install them.

Checking for the nova-agent and xe-guest-utilities

ps auxfwww | grep nova-agent
yum -qa xe-guest-utilities nova-agent
dpkg -l xe-guest-utilities nova-agent

Explanation and solution

Thanks for reaching out to us with your inquiry today. I’m glad to convey to you that I understand what the problem is with your cloud-server not being contactable.

Main reasons for breakage

The main reason why this is not working is most likely caused by some important pieces of software being missing. There is a piece of software called nova-agent, which is responsible for setting your cloud-servers IPV4 address, network subnet/mask, and ip routes, when it is first built. This is important, since the server image you built the server from, has different network details.

The rackspace build process giving networking detail to the VM is completely dependent on xe-guest-utilities and nova-agent

What has happened in this case, because the nova-agent wasn’t running on the cloud-server, the hypervisor software Rackspace use to automate cloud-server builds wasn’t able to contact the nova-agent running on your cloud-server, and therefore nova-agent wasn’t able to update the networking information. And hence, your not able to connect to it on it’s IPv4 address you are given at build time.

The steps to resolution: installing nova-agent and xen guest utilities
As such, nova-agent needs to be installed on the cloud-server you take the image from, it can be installed as follows:

https://community.rackspace.com/products/f/25/t/5694

Also nova-agent uses another piece of important software called xe-guest-utilities, or (Xen Tools) for your windows servers, this is an important ‘PV’ paravirtualization tools, responsible for seamless management of cloud-servers. Sorry that in this case it’s not working out seamlessly, but this can happen with images taken of servers which have had nova-agent disabled, uninstalled, or similar.

Upgrading the tools that nova-agent depends upon, can be installed by following the instructions at the following location:

https://support.rackspace.com/how-to/upgrade-citrix-xen-server-tools-for-windows-cloud-servers/

# Options of how to do this / Summary of Solution Steps

Naturally, you might be wondering how to achieve these changes, if you cannot RDP to the server. This is quite understandable, there are two ways to get this working;

Option 1) Manually install nova-agent on the current server you cannot access, then manually install the Xen Tools in the same way. This shall fix the OS on the server itself, and not the original image you built the server from. So it is important to create a new cloud-server image after performing these steps and us verifying tools + nova-agent installed correctly.

2) Manually install nova-agent on the source server you initially taken the image from, and install Xen Tools, then re-image the server, and then re-deploy. This should seamlesssly work each time on build with that image, provided the tools are installed. You will not need to recreate the image, since your fixing the problem on the cloud-server source that the original image was taken from.

I appreciate that these things are not 100% simple to get your head around and can be confusing for customers, I hope my explanation and summary makes this a little more painless to fix. Of course if you have additional questions, comments or concerns or don’t understand something I’ve said, please don’t hesitate to reach out to us, we are here to help!

Creating a proper Method of Retrieving, Sorting, and Parsing Rackspace CDN Access Logs

So, this has been rather a bane on the life which is lived as Adam Bull. Basically, a large customer of ours had 50+ CDN’s, and literally hundreds of gigabytes of Log Files. They were all in Rackspace Cloud Files, and the big question was ‘how do I know how busy my CDN is?’.

screen-shot-2016-11-07-at-12-41-30-pm

This is a remarkably good question, because actually, not many tools are provided here, and the customer will, much like on many other CDN services, have to download those logs, and then process them. But that is actually not easier either, and I spent a good few weeks (albeit when I had time), trying to figure out the best way to do this. I dabbled with using tree to display the most commonly used logs, I played with piwik, awstats, and many others such as goaccess, all to no avail, and even used a sophisticated AWK script from our good friends in Operations. No luck, nothing, do not pass go, or collect $200. So, I was actually forced to write something to try and achieve this, from start to finish. There are 3 problems.

1) how to easily obtain .CDN_ACCESS_LOGS from Rackspace Cloud Files to Cloud Server (or remote).
2) how to easily process these logs, in which format.
3) how to easily present these logs, using which application.

The first challenge was actually retrieving the files.

swiftly --verbose --eventlet --concurrency=100 get .CDN_ACCESS_LOGS --all-objects -o ./

Naturally to perform this step above, you will need a working, and setup swiftly environment. If you don’t know what swiftly, is or understand how to set up a swiftly envrionment, please see this article I wrote on the subject of deleting all files with swiftly (The howto explains the environment setup first! Just don’t follow the article to the end, and continue from here, once you’ve setup and installed swiftly)

Fore more info see:
https://community.rackspace.com/products/f/25/t/7190

Processing the Rackspace CDN Logs that we’ve downloaded, and organising them for further log processing
This required a lot more effort, and thought

The below script sits in the same folder as all of the containers

# ls -al 
total 196
drwxrwxr-x 36 root root  4096 Nov  7 12:33 .
drwxr-xr-x  6 root root  4096 Nov  7 12:06 ..
# used by my script
-rw-rw-r--  1 root root  1128 Nov  7 12:06 alldirs.txt

# CDN Log File containers as we downloaded them from swiftly Rackspace Cloud Files (.CDN_ACCESS_LOGS)
drwxrwxr-x  3 root root  4096 Oct 19 11:22 dev.demo.video.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:22 europe.assets.lon.tv
drwxrwxr-x  5 root root  4096 Oct 19 11:22 files.lon.cdn.lon.com
drwxrwxr-x  3 root root  4096 Oct 19 11:23 files.blah.cdn..com
drwxrwxr-x  5 root root  4096 Oct 19 11:24 files.demo.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:25 files.invesco.cdn..com
drwxrwxr-x  3 root root  4096 Oct 19 11:25 files.test.cdn..com
-rw-r--r--  1 root root   561 Nov  7 12:02 generate-report.sh
-rwxr-xr-x  1 root root  1414 Nov  7 12:15 logparser.sh

# Used by my script
drwxr-xr-x  2 root root  4096 Nov  7 12:06 parsed
drwxr-xr-x  2 root root  4096 Nov  7 12:33 parsed-combined
#!/bin/bash

# Author : Adam Bull
# Title: Rackspace CDN Log Parser
# Date: November 7th 2016

echo "Deleting previous jobs"
rm -rf parsed;
rm -rf parsed-combined

ls -ld */ | awk '{print $9}' | grep -v parsed > alldirs.txt


# Create Location for Combined File Listing for CDN LOGS
mkdir parsed

# Create Location for combined CDN or ACCESS LOGS
mkdir parsed-combined

# This just builds a list of the CDN Access Logs
echo "Building list of Downloaded .CDN_ACCESS_LOG Files"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
echo $folder
        echo "$m" | xargs -i find ./{} -type f -print > "parsed/$folder.log"
done < alldirs.txt

# This part cats the files and uses xargs to produce all the Log oiutput, before cut processing and redirecting to parsed-combined/$folder
echo "Combining .CDN_ACCESS_LOG Files for bulk processing and converting into NCSA format"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
cat "parsed/$folder.log" | xargs -i zcat {} | cut -d' ' -f1-10  > "parsed-combined/$folder"
done < alldirs.txt


# This part processes the Log files with Goaccess, generating HTML reports
echo "Generating Goaccess HTML Logs"
sleep 3
while read m; do
folder=$(echo "$m" | sed 's@/@@g')
goaccess -f "parsed-combined/$folder" -a -o "/var/www/html/$folder.html"
done < alldirs.txt

How to easily present these logs

I kind of deceived you with the last step. Actually, because I have already done it, with the above script. Though, you will naturally need to have an httpd installed, and a documentroot in /var/www/html, so make sure you install apache2:

yum install httpd awstats

De de de de de de da! da da!

screen-shot-2016-11-07-at-12-41-30-pm

Some little caveats:

Generating a master index.html file of all the sites


[root@cdn-log-parser-mother html]# pwd
/var/www/html
[root@cdn-log-parser-mother html]# ls -al | awk '{print $9}' | xargs -i echo " {}
" > index.html

I will expand the script to generate this automatically soon, but for now, leaving like this due to time constraints.

Checking a Website or Rackspace Load Balancers Supported SSL Ciphers

So, you may have recently had an audit performed, or have been warned about the dangers of SSLv3, poodle attack, heartbleed and etc. You want to understand exactly which ciphers your using on the Load Balancer, cloud-server, or dedicated server. It’s actually very easy to do this with nmap. Install it first, naturally.

# CentOS / RedHat
yum install nmap

# Debian / Ubuntu
apt-get install nmap

# Check for SSL ciphers

# nmap hostnamegoeshere.com --script ssl-enum-ciphers -p 443

Starting Nmap 6.47 ( http://nmap.org ) at 2016-10-11 09:12 UTC
Nmap scan report for 134.213.236.167
Host is up (0.0017s latency).
PORT STATE SERVICE
443/tcp open https
| ssl-enum-ciphers:
| SSLv3: No supported ciphers found
| TLSv1.0:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - strong
| compressors:
| NULL
| TLSv1.1:
| ciphers:
| TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA - strong
| TLS_RSA_WITH_3DES_EDE_CBC_SHA - strong
| TLS_RSA_WITH_AES_128_CBC_SHA - strong
| TLS_RSA_WITH_AES_256_CBC_SHA - strong
| compressors:
| NULL
| TLSv1.2: No supported ciphers found
|_ least strength: strong

Nmap done: 1 IP address (1 host up) scanned in 1.57 seconds

In this case we can see that only TLS v1.1 and TLS v1.0 are supported. No TLSv1.2 and no SSLv3.

Cheers &
Best wishes,
Adam

Using Sar to tell a story

So, a customer is experiencing slowness/sluggishness in their app. You know there is not issue with the hypervisor from instinct, but instinct isn’t enough. Using tools like xentop, sar, bwm-ng are critical parts of live and historical troubleshooting.

Sar can tell you a story, if you can ask the storyteller the write questions, or even better, pick up the book and read it properly. You’ll understand what the plot, scenario, situation and exactly how to proceed with troubleshooting by paying attention to these data and knowing which things to check under certain circumstances.

This article doesn’t go in depth to that, but it gives you a good reference of a variety of tests, the most important being, cpu usage, io usage, network usage, and load averages.

CPU Usage of all processors

# Grab details live
sar -u 1 3

# Use historical binary sar file
# sa10 means '10th day' of current month.
sar -u -f /var/log/sa/sa10 

CPU Usage of a particular Processor

sar -P ALL 1 1

‘-P 1’ means check only the 2nd Core. (Core numbers start from 0).

sar -P 1 1 5

The above command displays real time CPU usage for core number 1, every 1 second for 5 times.

Observing Changes in Memory over time

sar -r 1 3

The above command provides memory stats every 1 second for a total of 3 times.

Observing Swap usage over time

sar -S 1 5

The above command reports swap statistics every 1 seconds, a total 3 times.

Overall I/O activity

sar -b 1 3 

The above command checks every 1 seconds, 3 times.

Individual Block Device I/O Activities

This is a useful check for LUN , block devices and other specific mounts

sar -d 1 1 
sar -p d

DEV – indicates block device, i.e. sda, sda1, sdb1 etc.

Total Number processors created a second / Context switches

sar -w 1 3

Run Queue and Load Average

sar -q 1 3 

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. “1 3” reports for every 1 seconds a total of 3 times.

Report Network Statistics

sar -n KEYWORD

KEYWORDS Available;

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information. The output will be very long.

sar -n DEV 1 1

Specify Start Time

sar -q -f /var/log/sa/sa11 -s 11:00:00
sar -q -f /var/log/sa/sa11 -s 11:00:00 | head -n 10

Using TCP to ping test your Cloud server connectivity

So, you have probably heard that there are a variety of reasons why you shouldn’t use ICMP to test your service is operating normally. Mainly because of the way that ICMP is handled by routers. If you really want a representative view of the way that TCP packets, such as HTTP and HTTPS are performing in terms of packet loss (that is to say packets which do not arrive at their destination) , then hping is your friend.

You might be pinging a cloud-server that is not replying. You might think it’s down. But what if the firewall is simply dropping ICMP echo requests coming in on that port? Indeed.

Enter hping.

# hping -S -p 80 google.com
HPING google.com (eth0 74.125.136.102): S set, 40 headers + 0 data bytes
len=46 ip=74.125.136.102 ttl=46 id=23970 sport=80 flags=SA seq=0 win=42780 rtt=13.8 ms
len=46 ip=74.125.136.102 ttl=47 id=37443 sport=80 flags=SA seq=1 win=42780 rtt=12.6 ms
len=46 ip=74.125.136.102 ttl=47 id=43654 sport=80 flags=SA seq=2 win=42780 rtt=12.0 ms
len=46 ip=74.125.136.102 ttl=47 id=37877 sport=80 flags=SA seq=3 win=42780 rtt=11.4 ms
len=46 ip=74.125.136.102 ttl=47 id=62433 sport=80 flags=SA seq=4 win=42780 rtt=13.3 ms
^C
--- google.com hping statistic ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 11.4/12.6/13.8 ms

In this case I tested with google.com. I’m actually surprised that more people don’t use hping, because, hping is awesome. It also makes quite a decent port scanner, were it not for the fact that the machine I tried to test that feature with buffer overflowed 😉 It’s a nice way to test a firewalled box, but more than that, it’s a more reliable test in my opinion.

Simple way to perform a body check on a website

So, I was testing with curl today and I know that it’s possible to direct to /dev/null to suppress the page. But that’s not very handy if you are checking whether html page loads, so I came up with some better body checks to use.

A Basic body check using wc -l to count the lines of the site

 time curl https://www.google.com/ > 1; echo "non zero indicates server up and served content of n lines"; cat 1 | wc -l
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  167k    0  167k    0     0  79771      0 --:--:--  0:00:02 --:--:-- 79756

real	0m2.162s
user	0m0.042s
sys	0m0.126s
non zero indicates server up and served content of n lines
2134

A body check for Google analytics

$ time curl https://www.groundworkjobs.com/ > 1; echo "Checking for google analytics html elements string"; cat 1 | grep "www.google-analytics.com/analytics.js"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  167k    0  167k    0     0  76143      0 --:--:--  0:00:02 --:--:-- 76152

real	0m2.265s
user	0m0.042s
sys	0m0.133s
Checking for google analytics html elements string
				})(window,document,'script','//www.google-analytics.com/analytics.js','ga');

Such commands might be useful when troubleshooting a cluster for instance, where one server shows more up to date versions, (different number of lines). There’s probably better way to do this with ls and awk and use the html filesize, since number of lines wouldn’t be so accurate.

Check Filesize from request

$ time curl https://www.groundworkjobs.com/ > 1; var=$(ls -al 1 | awk '{print $5}') ; echo "Page size is: $var kB"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  167k    0  167k    0     0  79467      0 --:--:--  0:00:02 --:--:-- 79461

real	0m2.170s
user	0m0.048s
sys	0m0.111s
Page size is: 171876 kB

Pretty simple.. but you could take the oneliner even further… populate a variable called $var with the filesize using ls and awk , and then use an if statement to check that var is not 0, indicating the page is answering positively, or alternatively not answering at all.

Check Filesize and populate a variable with the filesize, then validate variable

$ time curl https://www.groundworkjobs.com/ > 1; var=$(ls -al 1 | awk '{print $5}') ; echo "Page size is: $var kB"; if [ "$var" -gt 0 ] ; then echo "The filesize was greater than 0, which indicates box is up but may be giving an error page"; fi
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  167k    0  167k    0     0  78915      0 --:--:--  0:00:02 --:--:-- 78950

real	0m2.185s
user	0m0.041s
sys	0m0.132s
Page size is: 171876 kB
The filesize was greater than 0, which indicates box is up but may be giving an error page

The second exercise is not particularly useful or practical as a means of testing, since if the site was timing out the script would take ages to reply and make the whole test pointless, but as a learning exercise being able to assemble one liners on the fly like this is an enjoyable, rewarding and useful investment of time and effort. Understanding such things are the fundamentals of automating tasks. In this case with output filtering, variable creation, and subsequent validation logic. It’s a simple test, but the concept is exactly the same for any advanced automation procedure too.