Count number of IP’s over a given time period in Apache Log

Posted on 24th February 2017 by azio

So, a customer had an outage, and wasn’t sure what caused it. It looked like some IP’s were hammering the site, so I wrote this quite one liner just to sort the IP’s numerically, so that uniq -c can count the duplicate requests, this way we can count exactly how many times a given IP makes a request in any given minute or hour:

Any given minute

# grep '24/Feb/2017:10:03' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any given hour

# grep '24/Feb/2017:10:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given day

# grep '24/Feb/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given Month

# grep '/Feb/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given Year

# grep '/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any given year might cause dupes though, and I’m sure there is a better way of doing that which is more specific

Checking Webserver Logs and generating hit counts

Posted on 16th January 2017 by azio

So, I’ve been meaning to do this for a while. I’m sure your all familiar with this crappy oneliner where you’ll check the hits using wc -l against a webserver log for a given hour and use for i in seq or similar to get the last 10 minutes data, or the last 59 minutes. But getting hours and days is a bit harder. You need some additional nesting, and, it isn’t difficult for you to do at all!

for i in 0{1..9}; do echo "16/Jan/2017:06:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/soemsite.com-access_log | wc -l ; done

I improved this one, quite significantly by adding an additional for j, for the hour, and adding an additional 0 in front of {1..9}, this properly is matching the Apache2 log format and allows me to increment through all the hours of the day. to present ;-D All that is missing is some error checking when the last date in the file is, im thinking a tail and grep for the timecode from the log should be sufficient.

Here is the proud oneliner I made for this!

for j in 0{1..9}; do for i in 0{1..9} {10..59}; do echo "16/Jan/2017:$j:$i"; grep "16/Jan/2017:06:$i" /var/log/httpd/website.com-access_log | wc -l ; done; done

Tracing Down Network and Process Traffic Using Netfilter

Posted on 16th January 2017 by azio

Every now and then at Rackspace, as with any hosting provider. We do occasionally have issues where customers have left themselves open to attack. In such cases sometimes customers find their server is sending spam email, and is prone to other malware occurring on the Rackspace Network.

Due to AUP and other obligations, it can become a critical issue for both the uptime, and reputation of your site. In many cases, customers do not necessarily have forensic experience, and will struggle very hard to remove the malware. In some cases, the malware keeps on coming back, or, like in my customers case, you could see lots of extra network traffic still using tcpdump locally on the box.

Enter, netfilter, part of the Linux Kernel, and it is able, if you ask it, to track down where packets are coming from, on a process level. This is really handy if you have an active malware or spam process on your system, since you can find out exactly where it is, before doing more investigation. Such a method, also allows you to trace down any potential false positives, since the packet address is always included, you get a really nice overview.

To give you an idea, I needed to install a kernel with debuginfo, just to do this troubleshooting, however this depends on your distribution.

Updating your Kernel may be necessary to use netfilter debug

$yum history info 18

Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                               @base
    Installed     yum-3.4.3-150.el7.centos.noarch                        @base
    Installed     yum-plugin-auto-update-debug-info-1.1.31-40.el7.noarch @base
    Installed     yum-plugin-fastestmirror-1.1.31-40.el7.noarch          @base
Packages Altered:
    Updated kernel-debuginfo-4.4.40-202.el7.centos.x86_64               @base-debuginfo
    Update                   4.4.42-202.el7.centos.x86_64               @base-debuginfo
    Updated kernel-debuginfo-common-x86_64-4.4.40-202.el7.centos.x86_64 @base-debuginfo
    Update                                 4.4.42-202.el7.centos.x86_64 @base-debuginfo

You could use a similar process using netfilter.ip.local_in, I suspect.

The Script

#! /usr/bin/env stap

# Print a trace of threads sending IP packets (UDP or TCP) to a given
# destination port and/or address.  Default is unfiltered.

global the_dport = 0    # override with -G the_dport=53
global the_daddr = ""   # override with -G the_daddr=127.0.0.1

probe netfilter.ip.local_out {
    if ((the_dport == 0 || the_dport == dport) &&
        (the_daddr == "" || the_daddr == daddr))
            printf("%s[%d] sent packet to %s:%d\n", execname(), tid(), daddr, dport)
}

Executing the Script

[root@pirax-test-new hacked]# chmod +x dns_probe.sh
[root@pirax-test-new hacked]# ./dns_probe.sh
Missing separate debuginfos, use: debuginfo-install kernel-3.10.0-514.2.2.el7.x86_64
swapper/3[0] sent packet to 78.136.44.6:0
sshd[25421] sent packet to 134.1.1.1:55336
sshd[25421] sent packet to 134.1.1.1:55336
swapper/3[0] sent packet to 78.136.44.6:0

I was a little bit concerned about the above output, it looks like swapper with pid 3, is doing something it wouldn’t normally do. Upon further inspection though, we find it is just the outgoing cloud monitoring call;

# nslookup 78.136.44.6
Server:		83.138.151.81
Address:	83.138.151.81#53

Non-authoritative answer:
6.44.136.78.in-addr.arpa	name = collector-lon-78-136-44-6.monitoring.rackspacecloud.com.

Authoritative answers can be found from:

Upgrading PHP 5.3.29 to PHP 7 on Centos 6.8 Using Rackspace IUS Repo

Posted on 13th January 2017 by azio

These instructions only apply in specific cases. Specifically CentOS machines, running in the Rackspace Cloud, IUS the Rackspace provided repo, provides several things not usually available within the CentOS repo, without you manually compiling more recent versions. One of them is the latest version of PHP7.0 and PHP7.1.

I wanted to quickly document the process, since it is a relatively simple process, and, can actually be done without any maintenance window, if you know what your doing, with very minimal, (if any) disruption to running sites. an apachectl graceful, actually, should be enough. Since apachectl gracefully restarts apache httpd, the downtime you’ll see will be super minimal. Expect nobody to notice you upgraded to PHP7 if you do this right.

If you do this incorrectly, you will break the PHP installation, and worse, break all of the sites using mod_php. Lets take a look at the steps:

Step 1. Check available PHP modules provided by presently configured REPO

root@server3 ~]# yum search php7
Loaded plugins: fastestmirror, versionlock
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
============================================================================================================================= N/S Matched: php7 ==============================================================================================================================
php70u-debuginfo.x86_64 : Debug information for package php70u
php70u-ioncube-loader-debuginfo.x86_64 : Debug information for package php70u-ioncube-loader
php70u-pecl-amqp-debuginfo.x86_64 : Debug information for package php70u-pecl-amqp
php70u-pecl-apcu-debuginfo.x86_64 : Debug information for package php70u-pecl-apcu
php70u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php70u-pecl-igbinary
php70u-pecl-imagick-debuginfo.x86_64 : Debug information for package php70u-pecl-imagick
php70u-pecl-redis-debuginfo.x86_64 : Debug information for package php70u-pecl-redis
php70u-pecl-smbclient-debuginfo.x86_64 : Debug information for package php70u-pecl-smbclient
php70u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php70u-pecl-xdebug
php71u-debuginfo.x86_64 : Debug information for package php71u
php71u-pecl-apcu-debuginfo.x86_64 : Debug information for package php71u-pecl-apcu
php71u-pecl-igbinary-debuginfo.x86_64 : Debug information for package php71u-pecl-igbinary
php71u-pecl-redis-debuginfo.x86_64 : Debug information for package php71u-pecl-redis
php71u-pecl-xdebug-debuginfo.x86_64 : Debug information for package php71u-pecl-xdebug
sclo-php70-php-pecl-propro-devel.x86_64 : sclo-php70-php-pecl-propro developer files (header)
sclo-php70-php-pecl-raphf-devel.x86_64 : sclo-php70-php-pecl-raphf developer files (header)
uwsgi-plugin-php70u-debuginfo.x86_64 : Debug information for package uwsgi-plugin-php70u
mod_php70u.x86_64 : PHP module for the Apache HTTP Server
mod_php71u.x86_64 : PHP module for the Apache HTTP Server
php70u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php70u-cli.x86_64 : Command-line interface for PHP
php70u-common.x86_64 : Common files for PHP
php70u-dba.x86_64 : A database abstraction layer module for PHP applications
php70u-dbg.x86_64 : The interactive PHP debugger
php70u-devel.x86_64 : Files needed for building PHP extensions
php70u-embedded.x86_64 : PHP library for embedding in applications
php70u-enchant.x86_64 : Enchant spelling extension for PHP applications
php70u-fpm.x86_64 : PHP FastCGI Process Manager
php70u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php70u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php70u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php70u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php70u-imap.x86_64 : A module for PHP applications that use IMAP
php70u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php70u-intl.x86_64 : Internationalization extension for PHP applications
php70u-ioncube-loader.x86_64 : IonCube Loader provides PHP Modules to read IonCube Encoded Files
php70u-json.x86_64 : JavaScript Object Notation extension for PHP
php70u-ldap.x86_64 : A module for PHP applications that use LDAP
php70u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php70u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php70u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php70u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php70u-opcache.x86_64 : The Zend OPcache
php70u-pdo.x86_64 : A database access abstraction module for PHP applications
php70u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php70u-pear.noarch : PHP Extension and Application Repository framework
php70u-pecl-amqp.x86_64 : Communicate with any AMQP compliant server
php70u-pecl-apcu.x86_64 : APC User Cache
php70u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php70u-pecl-apcu-panel.noarch : APCu control panel
php70u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php70u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php70u-pecl-imagick.x86_64 : Provides a wrapper to the ImageMagick library
php70u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php70u-pecl-smbclient.x86_64 : PHP wrapper for libsmbclient
php70u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php70u-pgsql.x86_64 : A PostgreSQL database module for PHP
php70u-process.x86_64 : Modules for PHP script using system process interfaces
php70u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php70u-recode.x86_64 : A module for PHP applications for using the recode library
php70u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php70u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php70u-tidy.x86_64 : Standard PHP module provides tidy library support
php70u-xml.x86_64 : A module for PHP applications which use XML
php70u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
php71u-bcmath.x86_64 : A module for PHP applications for using the bcmath library
php71u-cli.x86_64 : Command-line interface for PHP
php71u-common.x86_64 : Common files for PHP
php71u-dba.x86_64 : A database abstraction layer module for PHP applications
php71u-dbg.x86_64 : The interactive PHP debugger
php71u-devel.x86_64 : Files needed for building PHP extensions
php71u-embedded.x86_64 : PHP library for embedding in applications
php71u-enchant.x86_64 : Enchant spelling extension for PHP applications
php71u-fpm.x86_64 : PHP FastCGI Process Manager
php71u-fpm-httpd.noarch : Apache HTTP Server configuration for PHP-FPM
php71u-fpm-nginx.noarch : Nginx configuration for PHP-FPM
php71u-gd.x86_64 : A module for PHP applications for using the gd graphics library
php71u-gmp.x86_64 : A module for PHP applications for using the GNU MP library
php71u-imap.x86_64 : A module for PHP applications that use IMAP
php71u-interbase.x86_64 : A module for PHP applications that use Interbase/Firebird databases
php71u-intl.x86_64 : Internationalization extension for PHP applications
php71u-json.x86_64 : JavaScript Object Notation extension for PHP
php71u-ldap.x86_64 : A module for PHP applications that use LDAP
php71u-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php71u-mcrypt.x86_64 : Standard PHP module provides mcrypt library support
php71u-mysqlnd.x86_64 : A module for PHP applications that use MySQL databases
php71u-odbc.x86_64 : A module for PHP applications that use ODBC databases
php71u-opcache.x86_64 : The Zend OPcache
php71u-pdo.x86_64 : A database access abstraction module for PHP applications
php71u-pdo-dblib.x86_64 : PDO driver Microsoft SQL Server and Sybase databases
php71u-pecl-apcu.x86_64 : APC User Cache
php71u-pecl-apcu-devel.x86_64 : APCu developer files (header)
php71u-pecl-apcu-panel.noarch : APCu control panel
php71u-pecl-igbinary.x86_64 : Replacement for the standard PHP serializer
php71u-pecl-igbinary-devel.x86_64 : Igbinary developer files (header)
php71u-pecl-redis.x86_64 : Extension for communicating with the Redis key-value store
php71u-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
php71u-pgsql.x86_64 : A PostgreSQL database module for PHP
php71u-process.x86_64 : Modules for PHP script using system process interfaces
php71u-pspell.x86_64 : A module for PHP applications for using pspell interfaces
php71u-recode.x86_64 : A module for PHP applications for using the recode library
php71u-snmp.x86_64 : A module for PHP applications that query SNMP-managed devices
php71u-soap.x86_64 : A module for PHP applications that use the SOAP protocol
php71u-tidy.x86_64 : Standard PHP module provides tidy library support
php71u-xml.x86_64 : A module for PHP applications which use XML
php71u-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
sclo-php70-php-pecl-apcu.x86_64 : APC User Cache
sclo-php70-php-pecl-apcu-bc.x86_64 : APCu Backwards Compatibility Module
sclo-php70-php-pecl-apcu-devel.x86_64 : APCu developer files (header)
sclo-php70-php-pecl-apfd.x86_64 : Always Populate Form Data
sclo-php70-php-pecl-http.x86_64 : Extended HTTP support
sclo-php70-php-pecl-http-devel.x86_64 : Extended HTTP support developer files (header)
sclo-php70-php-pecl-lzf.x86_64 : Extension to handle LZF de/compression
sclo-php70-php-pecl-mongodb.x86_64 : MongoDB driver for PHP
sclo-php70-php-pecl-propro.x86_64 : Property proxy
sclo-php70-php-pecl-raphf.x86_64 : Resource and persistent handles factory
sclo-php70-php-pecl-selinux.x86_64 : SELinux binding for PHP scripting language
sclo-php70-php-pecl-solr2.x86_64 : Object oriented API to Apache Solr
sclo-php70-php-pecl-uploadprogress.x86_64 : An extension to track progress of a file upload
sclo-php70-php-pecl-uuid.x86_64 : Universally Unique Identifier extension for PHP
sclo-php70-php-pecl-xattr.x86_64 : Extended attributes
sclo-php70-php-pecl-xdebug.x86_64 : PECL package for debugging PHP scripts
uwsgi-plugin-php70u.x86_64 : uWSGI - Plugin for PHP support

  Name and summary matches only, use "search all" for everything.

As we can see php7 is there. Great. But what about the php packages they have already? That’s coming up next.

Step 2. Check presence of plugin replace, we’ll use this to upgrade to mod_php70 once we’re ready

# Locate plugin replace is available
yum search yum-plugin-replace
# Install yum plugin replace if available (otherwise this will not work for you as easily)
yum install yum-plugin-replace

Step 3. Run a mock yum replace

# yum replace php53u --replace-with mod_php70u.x86_64
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00
Error: No Package Matching mod_php70u.x86_64
[root@server3 ~]# yum replace php53u --replace-with mod_php70u
Loaded plugins: fastestmirror, replace, versionlock
Replacing packages takes time, please be patient...
Loading mirror speeds from cached hostfile
drivesrvr                                                                                                                                                                                                                                              | 2.2 kB     00:00

WARNING: Unable to resolve all providers: ['config(php53u-common)', 'curl.so()(64bit)', 'fileinfo.so()(64bit)', 'json.so()(64bit)', 'phar.so()(64bit)', 'php-api', 'php-pecl(Fileinfo)', 'php-pecl(phar)', 'php-pecl(zip)', 'php-pecl-Fileinfo', 'php-pecl-phar', 'php-pecl-zip', 'php-zend-abi', 'php53(api)', 'php53(language)', 'php53(zend-abi)', 'php53-api', 'php53-bz2', 'php53-calendar', 'php53-common', 'php53-ctype', 'php53-curl', 'php53-date', 'php53-exif', 'php53-filter', 'php53-ftp', 'php53-gettext', 'php53-gmp', 'php53-hash', 'php53-iconv', 'php53-json', 'php53-libxml', 'php53-openssl', 'php53-pcre', 'php53-pecl(Fileinfo)', 'php53-pecl(json)', 'php53-pecl(phar)', 'php53-pecl(zip)', 'php53-pecl-Fileinfo', 'php53-pecl-json', 'php53-pecl-phar', 'php53-pecl-zip', 'php53-posix', 'php53-reflection', 'php53-session', 'php53-shmop', 'php53-simplexml', 'php53-sockets', 'php53-spl', 'php53-sqlite3', 'php53-sysvmsg', 'php53-sysvsem', 'php53-sysvshm', 'php53-tokenizer', 'php53-wddx', 'php53-zend-abi', 'php53-zip', 'php53-zlib', 'php53u(api)', 'php53u(language)', 'php53u(zend-abi)', 'php53u-api', 'php53u-bz2', 'php53u-calendar', 'php53u-ctype', 'php53u-curl', 'php53u-date', 'php53u-exif', 'php53u-fileinfo', 'php53u-filter', 'php53u-ftp', 'php53u-gettext', 'php53u-gmp', 'php53u-hash', 'php53u-iconv', 'php53u-json', 'php53u-libxml', 'php53u-openssl', 'php53u-pcre', 'php53u-pecl(Fileinfo)', 'php53u-pecl(json)', 'php53u-pecl(phar)', 'php53u-pecl(zip)', 'php53u-pecl-Fileinfo', 'php53u-pecl-json', 'php53u-pecl-phar', 'php53u-pecl-zip', 'php53u-posix', 'php53u-reflection', 'php53u-session', 'php53u-shmop', 'php53u-simplexml', 'php53u-sockets', 'php53u-spl', 'php53u-sqlite3', 'php53u-sysvmsg', 'php53u-sysvsem', 'php53u-sysvshm', 'php53u-tokenizer', 'php53u-wddx', 'php53u-zend-abi', 'php53u-zip', 'php53u-zlib', 'zip.so()(64bit)', 'php53u-common', 'php53u-common(x86-64)', 'php53-cgi', 'php53-cli', 'php53-pcntl', 'php53-readline', 'php53u-cgi', 'php53u-pcntl', 'php53u-readline', 'php53u-cli', 'php53u-cli(x86-64)', 'config(php53u-pdo)', 'pdo.so()(64bit)', 'pdo_sqlite.so()(64bit)', 'php53-pdo', 'php53-pdo-abi', 'php53-pdo_sqlite', 'php53u-pdo-abi', 'php53u-pdo', 'php53u-pdo(x86-64)', 'config(php53u-mysql)', 'mysql.so()(64bit)', 'mysqli.so()(64bit)', 'pdo_mysql.so()(64bit)', 'php-mysql', 'php53-mysql', 'php53-mysqli', 'php53u-mysqli', 'php53u-mysql', 'php53u-mysql(x86-64)', 'config(php53u)', 'libphp5.so()(64bit)', 'mod_php53u', 'php53', 'php53u', 'php53u(x86-64)', 'libphp5.so()(64bit)', 'php53-zts', 'php53u-zts', 'php53u-zts(x86-64)']

This may be normal depending on the package.  Continue? [y/N] y
Resolving Dependencies
--> Running transaction check
---> Package mod_php70u.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php53u.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-cli.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-common.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-mysql.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pdo.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php53u-pear.noarch 1:1.9.4-3.ius.centos6 will be erased
---> Package php53u-zts.x86_64 0:5.3.29-1.ius.centos6 will be erased
---> Package php70u-cli.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-common.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-gmp.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-json.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-mysqlnd.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pdo.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-pear.noarch 1:1.10.1-2.ius.centos6 will be installed
---> Package php70u-process.x86_64 0:7.0.14-3.ius.centos6 will be installed
---> Package php70u-xml.x86_64 0:7.0.14-3.ius.centos6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                             Arch                                                        Version                                                                      Repository                                                 Size
==============================================================================================================================================================================================================================================================================
Installing:
 mod_php70u                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       2.7 M
 php70u-cli                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       4.0 M
 php70u-common                                                       x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       1.1 M
 php70u-gmp                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        65 k
 php70u-json                                                         x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        62 k
 php70u-mysqlnd                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       221 k
 php70u-pdo                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       115 k
 php70u-pear                                                         noarch                                                      1:1.10.1-2.ius.centos6                                                       ius                                                       362 k
 php70u-process                                                      x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                        72 k
 php70u-xml                                                          x86_64                                                      7.0.14-3.ius.centos6                                                         ius                                                       183 k
Removing:
 php53u                                                              x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.4 M
 php53u-cli                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      7.9 M
 php53u-common                                                       x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      3.4 M
 php53u-mysql                                                        x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      219 k
 php53u-pdo                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      126 k
 php53u-pear                                                         noarch                                                      1:1.9.4-3.ius.centos6                                                        @ius                                                      2.2 M
 php53u-zts                                                          x86_64                                                      5.3.29-1.ius.centos6                                                         @ius                                                      4.6 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install      10 Package(s)
Remove        7 Package(s)

Total download size: 8.8 M
Is this ok [y/N]: N
Exiting on user Command
Your transaction was saved, rerun it with:
 yum load-transaction /tmp/yum_save_tx-2017-01-13-10-57L3T7JK.yumtx
You have mail in /var/spool/mail/root

Naturally, if you are satisfied that you do not need php53u-zts, the only php module which is not supported by PHP7, then you can proceed.

If you are wondering what ZTS is, The php-zts package contains a module for use with the Apache HTTP
Server which can operate under a threaded server processing model. (source pbone.net CentOS REPO)

ZTS is not required for MPM prefork, and is generally only used with MPM worker, afaik. So as long as your using prefork apache httpd your fine;

# apachectl -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

In our case prefork is being used, not worker. So I don’t think ZTS being missing is going to affect us. So we can proceed with typing ‘y’.

And’ thats pretty much how you upgrade to php7, it’s really easy with Rackspace IUS.

A Unique Situation for grep (finding the files with content matching a specific pattern Linux)

Posted on 13th January 2017 by azio

This article explains how to find all the files that have a specific text or pattern within them, this is the article you’ve been looking for!

So today, I was dealing with a customers server where he had tried to configure BASIC AUTH. I’d found the httpd.conf file for the specific site, but I couldn’t see which file had basic auth setup as wrong. To save me looking through hundreds of configurations (and also to save YOU from looking through hundreds of configuration files) for this specific pattern. Why not use grep to recursively search files for the pattern, and why not use -n to give the filename and line number of files which have text in that match this pattern.

I really enjoyed this oneliner, and been meaning to work to put something like this together, because this kind of issue comes up a lot, and this can save a lot of time!

 grep -rnw '/' -e "PermitRootLogin"

# OUTPUT looks like

/usr/share/vim/vim74/syntax/sshdconfig.vim:157:syn keyword sshdconfigKeyword PermitRootLogin
/usr/share/doc/openssh-5.3p1/README.platform:37:instead the PermitRootLogin setting in sshd_config is used.

The above searches recursively all files in the root filesystem ‘/’ looking for PermitRootLogin.

I wanted to find which .htaccess file was responsible so I ran;

# grep -rnw '/' -e "/path/to/.htpasswd'

# OUTPUT looks like
/var/www/vhosts/somesite.com/.htaccess:14:AuthUserFile /path/to/.htpasswd

Configuring SFTP without chroot (the easy way)

Posted on 20th December 2016 by azio

So, I wouldn’t normally recommend this to customers. However, there are secure ways to add SFTP access, without the SFTP subsystem having to be modified. It’s also possible to achieve similar setup in a location like /home/john/public_html.

Let’s assume that public_html and everything underneath it is chowned john:john. So john:john has all the access, and apache2 runs with it’s own gid;uid. This was a pretty strange setup, and you don’t see it every day. But actually, it allowed me to solve another problem that I’ve been seeing/seeing customers have for a long time. That problem is the problem of effectively and easily managing permissions. Once I figured this out it was a serious ‘aha!’ moment!. Here’s why.

Inside the /etc/group, we find the customers developer has done something tragic:

[root@web public_html]# cat /etc/group | grep apache
apache:x:48:john,bob

But fine.. we’ll run with it.

We can see all the files inside their /home/john/public_html , the sight is not good

]# ls -al 
total 232
drwxrwxr-x 27 john john  4096 Dec 20 15:56 .
drwxr-xr-x 12 john john  4096 Dec 15 11:08 ..
drwxrwxr-x 10 john john  4096 Dec 16 09:56 administrator
drwxrwxr-x  2 john john  4096 Dec 14 11:18 bin
drwxrwxr-x  4 john john  4096 Nov  2 15:05 build
-rw-rw-r--  1 john john   714 Nov  2 15:05 build.xml
drwxrwxr-x  3 john john  4096 Nov  2 15:05 c
drwxrwxr-x  3 john john 45056 Dec 20 13:09 cache
drwxrwxr-x  2 john john  4096 Dec 14 11:18 cli
drwxrwxr-x 32 john john  4096 Dec 14 11:18 components
-rw-rw-r--  1 john john  1863 Nov  2 15:05 configuration-live.php
-rw-r--r--  1 john john  3173 Dec 15 11:08 configuration.php
drwxrwxr-x  3 john john  4096 Nov  2 15:05 docs
drwxrwxr-x  8 john john  4096 Dec 16 17:17 .git
-rw-rw-r--  1 john john  1734 Dec 14 11:21 .gitignore

It gets worse..

# cat /etc/passwd | grep john
john:x:501:501::/home/john:/bin/sh

Now, adding an sftp user into this, might look like a nightmare, but actually with some retrospective thought it was really easy.

Solving this mess:

Install Scponly

yum install scponly

Create new ‘SFTP’ user:

scponlyuser:x:504:505::/home/john:/usr/bin/scponly

Create a password for user scponlyuser

 
passwd scponlyuser

Solution to john:john permissions

[root@web public_html]# cat /etc/group | grep john
apache:x:48:john,bob
john:x:501:scponlyuser

We simply make scponlyuser part of the john group by adding the second line there. That way, the scponlyuser will have read/write access to the same files as the shell user, without exposing any additional stuff.

This was a cool solution to fixing this customers insecure solution, that they wanted to keep it the way they had, and was also great way to add an sftp account without requiring root jail. Whether it’s better than the root jail, is really debatable, however scponly enforces that only this account can be used only for SCP, as well as achieving sftp user access, without a jail.

I was proud of this achievement.. goes to show Linux permissions are really more flexible than we can imagine. And, whether you really want to flex those permissions muscles though, should be of concern. I advised this customer to change this setup, remove the /bin/sh, among other things..

We finally test SFTP is working as expected with the new scponlyuser


sftp> rmdir test
sftp> get index.php
Fetching /home/john/public_html/index.php to index.php
/home/john/public_html/index.php                                                                                     100% 1420     1.4KB/s   00:00
sftp> put index.php
Uploading index.php to /home/john/public_html/index.php
index.php                                                                                                                100% 1420     1.4KB/s   00:00
sftp> mkdir test
sftp> rmdir test

Just replace ‘scponly’ with whatever username your setting up. The only part that you need to keep the ‘scponly’ bit, is /usr/bin/scponly, this is the environment logging into. Apologies that scponly is so similar to scponlyuser ;-D

scponlyuser:x:504:505::/home/john:/usr/bin/scponly

I was very pleased with this! Hope that you find this useful too!

Comparing Files on the internet or CDN with MD5 to determine if they present same content

Posted on 23rd November 2016 by azio

So, a customer today was having some issues with their CDN. They said that their SSL CDN was presenting a different image, than the HTTP CDN. So, I thought the best way to begin any troubleshooting process would firstly be to try and recreate those issues. To do that, I need a way to compare the files programmatically, enter md5sum a handly little shell application usually installed by default on most Linux OS.

[user@cbast3 ~]$ curl https://3485asd3jjc839c9d3-08e84cacaacfcebda9281e3a9724b749.ssl.cf3.rackcdn.com/companies/5825cb13f2e6c9632807d103/header.jpeg -o file ; cat file | md5sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  382k  100  382k    0     0  1726k      0 --:--:-- --:--:-- --:--:-- 1732k
e917a67bbe34d4eb2d4fe5a87ce90de0  -

[user@cbast3 ~]$ curl http://3485asd3jjc839c9d3-08e84cacaacfcebda9281e3a9724b749.r45.cf3.rackcdn.com/companies/5825cb13f2e6c9632807d103/header.jpeg -o file2 ; cat file2 | md5sum
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  382k  100  382k    0     0  2071k      0 --:--:-- --:--:-- --:--:-- 2081k
e917a67bbe34d4eb2d4fe5a87ce90de0  -

As we can see from the output of both, the md5sum (the hashing) of the two files is the same, this means there is a statistically very very very high chance the content is exactly the same, especially when passing several hundred characters or more. The hashing algorithm is combination based, so the more characters, the less likely same combination is of coming around twice!

In this case I was able to disprove the customers claim’s. Not because I wanted to, but because I wanted to solve their issue. These results show me, the issue must be, if it is with the CDN, with a local edgenode local to the customer having the issue. Since I am unable to recreate it from my location, it is therefore not unreasonable to assume that it is a client side issue, or a failure on our CDN edgenode side, local to the customer. That’s how I troubleshooted this, and quite happy with this one! Took about 2 minutes to do, and a few minutes to come up with. A quick and useful check indeed, which reduces the number of possibilities considerably in tracing down the issue!

Cheers &
Best wishes,
Adam

Please note the real CDN location has been altered for privacy reasons

How to limit the amount of memory httpd is using on CentOS 7 with Cgroups

Posted on 22nd November 2016 by azio

CentOS 7, introduced something called CGroups, or control groups which has been in the stable kernel since about 2006. The systemD unit is made of several parts, systemD unit resource controllers can be used to limit memory usage of a service, such as httpd control group in systemD.

# Set Memory Limits for a systemD unit
systemctl set-property httpd MemoryLimit=500MB

# Get Limits for a systemD Unit
systemctl show -p CPUShares 
systemctl show -p MemoryLimit

Please note that OS level support is not generally provided with managed infrastructure service level, however I wanted to help where I could hear, and it shouldn’t be that difficult because the new stuff introduced in SystemD and CGroups is much more powerful and convenient than using ulimit or similar.

Aaron Mehar’s CBS to VHD solution for Rackspace Cloud

Posted on 11th November 2016 by azio

Hey. So another one of my colleagues put together this really awesome article. Although I was aware this could be done, he’s done a really good job or putting together the procedures, of turning your CBS BFV (boot from (network) volume) disk into a VHD file.

Rackspace CBS disks works over iscsi and are presented via the network. The difference between instance store on the hypervisor, (utilized by cloud-server images), and the disk store on the CBS is that the CBS disk is not a VHD, but an disk presented over network via iscsi.

So, to take a VHD, or an equivalent cloud-server image snapshot, you need to image the disk manually, as well as convert it to VHD.

Taking an image of a volume is not possible, and would not be downloadable. However there are some workarounds that can be done.

*** Please NOTE ****
This is not supported, and we can not assist beyond these instructions. I could provide some clarity if required, however, my collegaues may not be able to help should I become unavailable.

If you just want the data, then you could just download the data to your local machine, however, if you a VHD to create a local VM, then the below instructions will achieve this.

Steps

Please take special care, making a mistake working with partitioner can wipe all your data

1. Shutdown the server
2. Clone the disk, by Starting a volume clone and start the server back up.
3. Attach the newly created clone to the server
4. create another new CBS volume of a slightly larger size (+5GB is OK)

Now that is done, we can image the disk. You will need to ensure you have the corrects disks. The second disk with data should be xvbd and the new CBS should be xvdc

Create partition and filesystem for xvdbc. Please see this guide: https://support.rackspace.com/how-to/prepare-your-cloud-block-storage-volume/

the image xvdb to xvdc

   dd if=/dev/xvdb of=/mnt/cbsvolume1/myimage.dd

The download the image to your workstation, and install VirtualBox, and run the below command

   VBoxManage convertfromraw myfile.dd myfile.vhd --format VHD

Please take special care, making a mistake working with partitioner can wipe all your data

Fixing nova-agent bugs caused by yum update

Posted on 11th November 2016 by azio

1. Download newest version of nova-agent from the github repo to the server you want to upgrade : https://github.com/rackerlabs/openstack-guest-agents-unix/releases

For this instance I used “nova-agent-1.39-1.x86_64.rpm” since it’s CentOS / Redhat based

2. Stop current nova-agent service

service nova-agent stop

3. Remove current in place nova-agent. I found the easiest way to do this is to just remove the entire contents of it’s directory.

rm -rf /usr/share/nova-agent/*

4. Install the new nova-agent with the RPM

rpm -ivh --nosignature nova-agent-1.39-1.x86_64.rpm

5. Start the new nova-agent service

service nova-agent start (might need to use systemd for CentOS/RHEL 7 and above)
# ie 
systemctl enable nova-agent
systemctl start nova-agent

6. Issue a networking reset to verify it is working. Check logs and verify that you see this message : ‘resetnetwork’ completed with code ‘0’, message ”

uuid=$(uuidgen)
xenstore-write data/host/$uuid '{"name":"resetnetwork","value":""}'

tail -20 /var/log/nova-agent.log

That’s it! Once you’ve done that you should reboot to verify that the nova-agent comes up on boot but otherwise nothing else is needed. Hope it helps!

Thanks to Sean from Rackspace for this.. you rock dudery.

haxed.me.uk

a place of infrastructure delights

Category Archives: Management Tools