Reverting a yum transaction & controlling auto-updates for yum-cron with excludes

Hey, so a customer was running yum-cron, the Redhat version of Canoninical’s unattended upgrades. An auto update for a package ran, which actually broke their backend.

# yum history info 172
Loaded plugins: replace, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
Transaction ID : 172
Begin time     : Sat May  6 05:45:29 2017
Begin rpmdb    : 742:11fcb243cc5701b9d2293d90cb4161e5edc34bb8
End time       :            05:45:31 2017 (2 seconds)
End rpmdb      : 742:0afbbf517315f18985b2d01d4c2e5250caf0afb5
User           : root <root>
Return-Code    : Success
Transaction performed with:
    Installed     rpm-4.11.3-21.el7.x86_64                @rhel-x86_64-server-7
    Installed     yum-3.4.3-150.el7.noarch                @rhel-x86_64-server-7
    Installed     yum-metadata-parser-1.1.4-10.el7.x86_64 @anaconda/7.1
    Installed     yum-rhn-plugin-2.0.1-6.1.el7_3.noarch   @rhel-x86_64-server-7
Packages Altered:
    Updated php56u-pecl-xdebug-2.5.1-1.ius.el7.x86_64 @rackspace-rhel-x86_64-server-7-ius
    Update                     2.5.3-1.ius.el7.x86_64 @rackspace-rhel-x86_64-server-7-ius
history info

The customer asked us to reverse the transaction, so I did, this was quite simple to do;

[root@web user]# yum history undo 172
Loaded plugins: replace, rhnplugin
This system is receiving updates from RHN Classic or Red Hat Satellite.
Undoing transaction 172, from Sat May  6 05:45:29 2017
    Updated php56u-pecl-xdebug-2.5.1-1.ius.el7.x86_64 @rackspace-rhel-x86_64-server-7-ius
    Update                     2.5.3-1.ius.el7.x86_64 @rackspace-rhel-x86_64-server-7-ius
Resolving Dependencies
--> Running transaction check
---> Package php56u-pecl-xdebug.x86_64 0:2.5.1-1.ius.el7 will be a downgrade
---> Package php56u-pecl-xdebug.x86_64 0:2.5.3-1.ius.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

 Package                                                  Arch                                         Version                                               Repository                                                                Size
 php56u-pecl-xdebug                                       x86_64                                       2.5.1-1.ius.el7                                       rackspace-rhel-x86_64-server-7-ius                                       205 k

Transaction Summary
Downgrade  1 Package

Total download size: 205 k
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
php56u-pecl-xdebug-2.5.1-1.ius.el7.x86_64.rpm                                                                                                                                                                        | 205 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : php56u-pecl-xdebug-2.5.1-1.ius.el7.x86_64                                                                                                                                                                                1/2
  Cleanup    : php56u-pecl-xdebug-2.5.3-1.ius.el7.x86_64                                                                                                                                                                                2/2
  Verifying  : php56u-pecl-xdebug-2.5.1-1.ius.el7.x86_64                                                                                                                                                                                1/2
  Verifying  : php56u-pecl-xdebug-2.5.3-1.ius.el7.x86_64                                                                                                                                                                                2/2

  php56u-pecl-xdebug.x86_64 0:2.5.3-1.ius.el7

  php56u-pecl-xdebug.x86_64 0:2.5.1-1.ius.el7


In CentOS 7 and I believe RedHat RHEL 7 as well, if you don’t want to disable yum-cron altogether by running a yum remove yum-cron, you could exclude the specific package, or use wildcards to exclude all of them like php* , http*, etc.

vi /etc/yum/yum-cron.conf.

If you wish to exclude some packages from auto-update mechanism, you’ll have to add an exclude line, at the bottom of the file, in the base section.

exclude = kernel* php* httpd*

etc, I hope that this is of some assist,

cheers &
Best wishes,

Magento Rewrite Woes … really woes

I had a customer this week that had some terrible rewrite woes with their magento site. They knew that a whole ton of their images were getting 404’s most likely because rewrite wasn’t getting to the correct filesystem path that the file resided. This was due to their cache being broken, and their second developer not creating proper rewrite rule.

As a sysadmin our job is not a development role, we are a support role, and in order to enable the developer to fix the problem, the developer needs to be able to see exactly what it is, enter the sysads task. I wrote this really ghetto script, which essentially hunts in the nginx error log for requests that failed with no such file, and then qualifies them by grepping for jpg file types. This is not a perfect way of doing it, however, it is really effective at identifying the broken links.

Then I have a seperate routine that strips the each of the file uri’s down to the filename, and locates the file on the filesystem, and matches the filename on the filesystem that the rewrite should be going to, as well as the incorrect path that the rewrite is presently putting the url to. See the script below:


# Author: Adam Bull
# Company: Rackspace LTD, Hayes
# Purpose:
#          This customer has a difficulty with nginx rewriting to the incorrect file
# this script goes into the nginx error.log and finds all the images which have the broken rewrite rules
# then after it has identified the broken rewrite rule files, it searches for the correct file on the filesystem
# then correlates it with the necessary rewrite rule that is required
# this could potentially be used for in-place upgrade by developers
# to ensure that website has proper redirects in the case of bugs with the ones which exist.

# This script will effectively find all 404's and give necessary information for forming a rewrite rule, i.e. the request url, from nginx error.log vs the actual filesystem location
# on the hard disk that the request needs to go to, but is not being rewritten to file path correctly already

# that way this data could be used to create rewrite rules programmatically, potentially
# This is a work in progress

# These are used for display output
cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print "URL Request:",$21,"\nFilesystem destination missing:",$7"\n"}'
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print "URL Request:",$21,"\nFilesystem destination detected missing:",$7"\n"}'

# These below are used for variable population for locating actual file paths of missing files needed to determine the proper rewrite path destination (which is missing)
# we qualify this with only *.jpg files

cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print $7}' | sed 's/\"//g' |  sed 's/.*\///' | grep jpg > lost.txt
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print $7}' | sed 's/\"//g' |  sed 's/.*\///' | grep jpg >> lost.txt

cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print "",$21}' | sed 's/\"//g' | grep jpg > lostfullurl.txt
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print "",$21}' | sed 's/\"//g' | grep jpg >> lostfullurl.txt

# The below section is used for finding the lost files on filesystem and pairing them together in variable pairs
# for programmatic usage for rewrite rules

while true
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  printf '\n\n'
  printf 'Found a broken link getting a 404 at : %s\n'
  printf "$f1\n"
  printf 'Locating the correct link of the file on the filesystem: %s\n'
        find /var/www/magento | grep $f2
done 3<lostfullurl.txt 4<lost.txt

I was particularly proud of the last section which uses a ‘dual loop for two input files’ in a single while statement, allowing me to achieve the descriptions above.

Output is in the form of:

Found a broken link getting a 404 at :
Locating the correct link of the file on the filesystem:

As you can see the path is different on the filesystem to the url that the rewrite is putting the request to, hence the 404 this customer is getting.

This could be a really useful script, and, I see no reason why the script could not generate the rewrite rules programatically from the 404 failures it finds, it could actually create rules that are necessary to fix the problem. Now, this is not an ideal fix, however the script will allow you to have an overview either to fix this properly as a developer, or as a sysadmin to patch up with new rewrite rules.

I’m really proud of this one, even though not everyone may see a use for it. There really really is, and this customer is stoked, think of it like this, how can a developer fix it if he doesn’t have a clear idea of the things that are broken, and this is the sysads job,

Cheers &
Best wishes,

Recovering Corrupt RPM DB

I had a support specialist today that had an open yum task they couldn’t kill gracefully, after kill -9, this was happening

[root@mybox home]# yum list | grep -i xml
rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv-&gt;failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
Error: rpmdb open failed
[root@mybox home]#

The solution to fix this is to manually erase the yumdb and to manually rebuild it;

[root@mybox home]# rm -f /var/lib/rpm/__*
[root@mybox home]# rpm --rebuilddb

Increasing the Limits of PHP-FPM

It’s important to know how to increase the limits for php-fpm www pools, or any other named alias pools you might have setup.

You might see an error like

tail -f /var/log/php7.1-fpm.log
[24-Apr-2017 11:23:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 11 total


[24-Apr-2017 10:51:38] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

The solution is quite simple, we just need to go in and edit the php fpm configuration on the server and increase these values to safe ones that is supported by available RAM.

pm.max_children = 15

; The number of child processes created on startup.
; Note: Used only when pm is set to 'dynamic'
; Default Value: min_spare_servers + (max_spare_servers - min_spare_servers) / 2
pm.start_servers = 2

; The desired minimum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.min_spare_servers = 1

; The desired maximum number of idle server processes.
; Note: Used only when pm is set to 'dynamic'
; Note: Mandatory when pm is set to 'dynamic'
pm.max_spare_servers = 8

Then monitor the site with

tail -f /var/log/php7.1-fpm.log

To ensure no further limits are being hit.

Obviously if you are using different version of fpm your log location might be different.

Finding all of the php.ini Files and finding what the memory_limit in them is

Simple as it says.

root@aea-php-slave:~# grep -rnw '/etc' -e 'memory_limit'
/etc/php/5.6/cli/php.ini:393:memory_limit = -1
/etc/php/5.6/apache2/php.ini:393:memory_limit = 128M
/etc/php5/fpm/php.ini:406:memory_limit = 128M
/etc/php5/fpm/pool.d/www.conf.disabled:392:;php_admin_value[memory_limit] = 32M
/etc/php5/fpm/pool.d/site.conf:392:;php_admin_value[memory_limit] = 32M
/etc/php5/cli/php.ini:406:memory_limit = -1
/etc/php5/apache2/php.ini:406:memory_limit = 128M


Setting 404 Error page in nginx

It’s easy to setup an custom error page in nginx. Just create a file in your documentroot like /404.html so if your documentroot is /var/www/html create a file called 404.html, then go into your /etc/nginx/nginx.conf, or /etc/nginx/conf.d/mysite.conf, and add this in your configuration between server { } directive.

Remember if your running https and http you will need the directive for each of them.

error_page 404 /404.html;

It’s possible to define a really custom location securely;

        error_page 404 /cust_404.html;
        location = /cust_404.html {
                root /usr/share/nginx/html;

Just make sure you don’t have a filename called 404.html in /404.html as I’m not sure this would work otherwise.

Plesk website running FastCGI Timeout Gateway Errors and Slow Loads

So my friend Paul shown me how to troubleshoot fastCGI on plesk boxes, very easy..

# pstree | grep cgi
     |       `-httpd---20*[php-cgi]

We can see that 20 php-cgi processes run. If we check the maximum in the configuration file set for fastCGI of Apache2..

# cat /etc/httpd/conf.d/fcgid.conf  | grep MaxProc
  FcgidMaxProcesses 20

We see 20 is the maximum, so it’s definitely hitting the FastCGI limit, we need to increase the limits, so we just edit the file and increase the limits for that variable;

vi /etc/httpd/conf.d/fcgid.conf

echo "FcgidMaxProcesses 50" >> /etc/httpd/conf.d/fcgid.conf

Setting a seperate memory limit for PhpMyAdmin to the rest of the sites

A common issue I see Rackspace customers with is there PhpMyAdmin not having enough memory, often I ‘ll see countless tickets where the memory_limit is increased for phpmyadmin, and when one of their virtualhosts falls over, it is then decreased for all of the sites, until someone wants to use phpmyadmin again.

not very good really is it? Actually, fixing this is quite easy. Lets provide a php.ini for phpmyadmin that only phpmyadmin uses;

# Copy original php configuration
cp /etc/php.ini /usr/share/phpMyAdmin/php.ini

# Modify /usr/share/phpMyAdmin/php.ini so that the following variable is set as a higher value
memory_limit = 256M

Naturally if you now goto the you’ll see a nice php file waiting for you… not good… we need to protect the php.ini file as it can expose stuff we don’t want to be seen; lets make it as hard to find out the server configuration and hide php.ini altogether.

# File to edit may differ but it can be any file inside conf.d make one if you like call it phpini.conf or something
vi /etc/httpd/conf.d/php.conf
<Files php.ini>
          Order allow,deny
          Deny from all

Dont’t forget the most important step

# Check apache syntax
apachectl -t

# Restart the apache process
apachectl graceful

Another pretty simples thing to do. That isn’t pretty simple until you do it.

Controlling MySQL Queries, and killing if necessary.

Today we had a customer call in who had a problem with their magento site. What had happened was they had reached the maximum number of connections for their MySQL.

 250481 | someuser | localhost:7777 | somewebsite | Query   | 2464 | Writing to net               | SELECT /*!40001 SQL_NO_CACHE */ * FROM `mg_sales_flat_quote_address`                                 

| 250486 | someuser | localhost       | somewebsite | Query   | 2459 | Waiting for table level lock | INSERT INTO `mg_sales_flat_quote_address` (`quote_id`, `created_at`, `updated_at`, `customer_id`, `a |         0 |             0 |         0 |

.. and a whole load more waiting for table level lock were underneath

Solution is simple;

kill 250481

250481 is the processid of the select query causing these issues. I suspect it will go away, eventually but in the meantime, want to make sure the max connections don’t get reached, otherwise website will be down again!

How Rackspace CDN works and what are the requirements?

The Rackspace CDN isn’t too complicated to setup. If you wish to configure the CDN with an origin dedicated server, there is really only one main requirement;

For example, if the CDN domain rackspace gives you for use with the CDN product is, you will be required to update the website configuration (origin) for Apache on this server.

i.e. you need to include the CDN domain as ServerAlias for Apache or as Server_Name for Nginx. Then that server alias/virtualhost receives and handles the request for the CDN.

Basically for CDN request comes in to rackspace domain, and if there is no cached version on a local edgenode, the CDN makes a call to your origin server IP configured, sending a HTTP header hostname. This is indeed why you need to setup a serveralias for the hostname, once it is created.

Provided that files can be accessed on the IP address of your origin dedicated server. i.e.

When request comes into it will if not cached try the origin server.

If you wish, you can add an additional CNAME of your own for your own domain.

Example only:
i.e. CNAME

This allows customer to make request to which forwards to Rackspace CDN, and if it has no cached file, it will forward to origin server the request, and cache the file around the edge nodes. The edge nodes are transparent, and the only thing you really need to worry about is configuring the virtualhost correctly.

The only reason you require the virtualhost, is because the CDN uses hostheader to allow your server to identify the virtualhost serving CDN. This naturally allows you to serve CDN on the same IP address. Naturally, if you intend to use SSL as well you may wish to consider a dedicated IP and find this community article of use.

Rackspace can help you configure the virtualhost and potentially may even be able to help you configure CDN product with your dedicated server origin as well.

For pricing, please see the CDN calculator at;

Cheers &
Best wishes,