Adding a User with Sudo Access using visudo

Use the command visudo to access the /etc/sudoers file.

visudo

Uncomment this line:

## Allows people in group wheel to run all commands
# %wheel        ALL=(ALL)       ALL

So it looks like:

## Allows people in group wheel to run all commands
 %wheel        ALL=(ALL)       ALL

Save the file then

Run this command for your user

usermod -aG wheel usernameforsudoaccesshere

Your done.

But test it

su usernamewithsudoaccess
sudo yum history

Any root only command is a good enough test for this. The command should run succesfully after re-providing your users password for sudo access.

Converting a QEMU qcow2 cloud server image to an native disk img and putting on physical disk

Got this question at work a lot. Thought I’d finally get around to putting it down since it’s came up for me. I’ve got a virtual machine using virtio passthrough for my pcie, and I found actually that disk access via the qcow2 is pretty naff.

sudo apt-get install qemu-kvm

qemu-img convert windows10cloudimage.qcow2 -O raw diskimage.img

dd if=/path/to/windos10cloudimage.qcow2 of=/dev/sdc2

Please note in my case the physical partition I’d made was sdc2. I’d actually resized another 5TB disk I have in my system using gparted. Just so I can attach a physical partition with libvirtd. Evidently though libvirtd-manager doesn’t allow this business so I have to edit the xlm file in /etc/qemu/windows10.xml .

 

root@adam:/etc/libvirt/qemu# virsh  define /etc/libvirt/qemu/win10-uefi.xml 
Domain win10-uefi defined from /etc/libvirt/qemu/win10-uefi.xml
root@adam:/etc/libvirt/qemu# virt-manager

yeah baby!


You could alternatively do it all in one like below, though you may desire a copy of the img file as well as putting it to the disk.

qemu-img convert windows10.qcow2 -O raw /dev/sdc

Configuring PCIE Passthrough and SLI/Crossfire on Ubuntu 16.04 step by step (for real)

GREETINGS! Welcome to this unusual Howto. So, thinking about setting up some PCIE Passthrough on your Linux box, so you can feel less embarrassed about needing windows to run your steam-games? Look no further!

Looking to setup SLI on your Linux Ubuntu Windows virtual machine? Look no further!!!

Are you looking to overcome some of the mean issues that are presented by devices with the same ID? (This is a big problem guys sometimes when passing through devices, if the device ID’s are the same you can also get problems).

Are you looking to discover why USB passthrough doesn’t work properly (or at all?) for your mouse/keyboard/usb audio? Look no further!! It’s quite common apparently that Ubuntu’s App armor, either, without being insecurely disabled, or a secure rule added, will prevent you from using libvirtd usb passthrough, due to the way the apparmor prevents sharing between applications that are unsupported. Whilst apparmor goes a little bit beyond the scope of this tutorial. Setting up this basic (starter) VM with 2 RX 580, is really a synch once you have it down. There are some important considerations before beginning though.

1. You will need an IGFX (integrated gfx) on your motherboard. Why you ask? Thats what Ubuntu is going to use. Whilst we will pass to pci-stub (and possibly vtio too I believe in some setups). VTIO pci passthrough is more modern the pci-stub passthrough I use. Though, when mining with the 2 rx 580 sapphire nitro I have within the virtual machine I got 22MH/s, knowing that these cards run stock unvirtualized in this machine at 23MH/s. I was able to calculate that PCIE passthrough, when configured correctly really is capable of 95% efficiency. To ensure that the RX580 devices are properly used, we must set in our BIOS, IGFX.

Step 1. Prepare BIOS display adapter/primary order


The first thing you need to do for Ubuntu PCIE passthrough is set the initial display output,
Change this from PCIE 1 slot (or whatever is set) to IGFX. Some bios might have different menu for enabling the igfx. What this setting does is toggle the integrated graphics as the primary graphics adapter when passed via the BIOS. This will actually allow you to start Ubuntu 16.04 without the amd_driver used for the RX580 cards. Also, because X on ubuntu will automatically bind to the IGFX, you don’t need to update your Ubuntu configuration to achieve this wondrous task of passthrough.

Though, if you have windows 7 or 10 on dual boot, Windows may prefer you to set the Initial display output back to PCIE 1 or whatever it was before. So take special care to note (or save) your bios configuration before making these changes.

I have 2 profiles ‘virtualized igfx for ubuntu’ and ‘pcie slot for windows’ are the name of my configs for logical purposes!

Step 2. Understanding IGFX, and the nature of the VT-d and VTIO and what it needs to work

Now you’ve setup the IGFX you can boot into Ubuntu normally. Installing Linux goes kind of below the scope of this tutorial. If you do not know how to configure Linux. Please find another howto and install Ubuntu 16.04 Linux first, then continue onto this step.


Be sure to ensure that VT-d or ‘Virtualization’ is also enabled in your BIOS, without it, nothing works, and you will cry with your kernel config wondering why.

Ubuntu will not be using the RX580 card, and the drivers may or may not be loaded in the kernel, depending on whether the OS is using the cards. In my situation, I had already disabled the rx580 driver in the kernel, and set the blacklist in modprobe.d accordingly, as well as ensured the module order list was correct, to allow me to achieve the ‘pass’ of the RX580 PCI interupts to the pci-stub kernel driver. The pci-stub kernel driver will then in turn pass that thru the VT-D (virtualisation) and VTIO (virtual input/output) bus supported by your CPU, and this is how the graphics cards get passed to the pass-through virtual machine. Therefore, if you haven’t done so already. Make sure that VT-d or ‘virtualization’ is enabled in your BIOS. Please note that some cpu do not support VTd or VT-x extensions. However if you have intel i3, i5, i7 etc, or any modern chip, you should definitely have this. Some motherboards have limited virtualization settings, so ensure you’ve properly looked into step 1 before going further here. It’s very important everything is just-right. There are a lot of turning wheels in pci-passthrough, but it’s remarkably simple once you get the basics.

Step 3: Final preparations for Linux Config. Stuff you may need

Remember GPU passthrough relies on specific technology. Providing your CPU, BIOS and PCIE graphics card, such as the RX580 I use in this tutorial are modern. You should be OK.

If in the BIOS you do not see something similar to “VT-d” (Intel) or “AMD Virtualization” (AMD), you should probably consult your BIOS manual.

Once you know the hardware is suitable, you’ll probably want to consider these additional factors, just to make sure your not wasting your time with this howto;

  • You should have at least 2 graphics cards (i.e. one onboard gpu on the motherboard and another discrete graphics card in one of the PCIE slots of the motherboard).


I’m using the RX580 Sapphire NITRO+ 8GB Graphics cards, 2 of them actually, + IGFX. Don’t worry this tutorial shows you how to configure, 1 or 2 PCIE passthroughs, and in SLI or Crossfire too! How awessseom!!?

  • Consider a second monitor, it’s quite annoying using the same monitor and having to switch from my DVI to HDMI, all the time. (DVI being the IGFX port connected to the monitors DVI input and HDMI being the input on my monitor plugged into the discrete PCIE RX580 graphics card(s).)
  • You do need a copy of Windows. I downloaded the iso available here which you can use without a key temporarily.
  • Although not 100% necessary, it’s much better to use a dedicated additional SSD disk with your virtual machine, VTIO is capable of passing through the physical hard disk, instead of relying on the vmdk or qcow raw file, which has the limitation of being a chained vdi. This has serious performance implications in terms of read/writes from the disk. Obviously the advantage of also passing an SSD to the machine is the speed optimization from the kernel being able to pass directly between the PCI and north bridge bus of the motherboard, directly thru the cpu and to the virtual machine, without any kind of abstraction being required (as was the case before PCIE passthrough virtualization type devices i.e. cirrus/vga etc).
  • In my example I’m just using a Qcow2 file, but adding SSD is so easy to do, you’ll have no difficulty if your capable of doing the PCIE passthrough kernel config.

MY SYSTEM

  • Graphics: AMD Sapphire ‘Radeon’ RX580 8GB GDDR5 NITRO+
  • CPU: Intel Core i7-6700 4GHz
  • Motherboard: Gigabyte Z170X ultra gaming
  • 32GB System RAM
  • USB Audio device if you do not want to use your monitor’s speakers for HDMI OUTPUT (yes the virtual machine can use the inbuilt audio of your graphics card, and yes, your graphics card obviously has inbuilt audio for HDMI standard which includes sound with audio bus. Link of the device I used below.
  • https://www.amazon.co.uk/Plugable-Headphone-Microphone-Aluminum-Compatibility-Black/dp/B00NMXY2MO/ref=sr_1_7?s=computers&ie=UTF8&qid=1517989287&sr=1-7&keywords=usb+audio

Step 4. Checking it all checks out

You might not see this output straight away, and you may need to change your module order later in step 5 to see some of these confirmation checks show good. Don’t freak out. So many moving wheels, arrghh.

Check Motherboard is playing the Virtualization extensions

dmesg | grep -e "Directed I/O"

For AMD Systems

dmesg | grep AMD-Vi

Output should be like:

AMD-Vi: Enabling IOMMU at 0000:00:00.2 cap 0x40
AMD-Vi: Lazy IO/TLB flushing enabled
AMD-Vi: Initialized for Passthrough Mode

HURRAH! We’re probably OK. Now lets get down to it.

Step 5. Locating my PCIE Graphics Cards

lspci | grep VGA

In my setup you can see there are 3 Adapters in total. The Intel Corporation Sky Lake integrated graphics adapter, is the motherboard hdmi output VGA. The other 2 VGA below are the 2 rx580 I have in both PCIE slots 1 and 2. As denoted by their PCI bus id 01:00.0 and 02:00.0 respectively. These are the hardware addresses used by the Local Linux ‘HOST OS’. That is to say, the hypervisor machine, the machine which is not virtualized hosting our virtual machine, is the one which knows about these addresses, the virtual machine knows nothing though 🙁

Step 6 Instructing the Linux Kernel

We need to tell Linux what to do , so the pci-stub kernel driver module is able to do it’s heavy lifting and just redirect command struct to the pcie ports irrelevant to the Ubuntu OS. If you kind of think about how awesome this is and how fancy it is have the host operating system pass a device to a virtual OS and BIOS as if the graphics card was running on a host windows OS. And 95% efficiency too (see my mining tests later on!). Simply amazing. Lets tell linux about it;

Step 6A: identifying the vendor ID of the RX580 Graphics cards is necessary to pass to pci-stub

lspci -nn | grep 01:00
lspci -nn | grep 02:00


Just showing the ID is the same for my two RX580 cards, mainly because they both have exactly the same model number. I found other PCIE passthrough guides which basically said oh you can’t do it properly because VTIO blah blah blah can’t do it. Well it can actually and I did achieve working RX580 x 2 in my windows guest. I guess this could be useful for people that wanted to virtualization their mining operations.  Note our PCIE RX580 graphics cards have an AUDIO DEVICE!!!!!

It’s a real pain having to drive out somewhere, the idea you could create this on a virtualization layer that was rock solid stable like Ubuntu has to be good. At least if you weren’t using something like EthOS already. Which by the way, is very good if your into your mining. Probably irrelevant though. Most people just want to play their games, and me too!!

We’ll come back to the hardware ID’s in a sec.

STEP 7A: Use Grub Bootloader to blacklist kernel drivers (we don’t want our Host Ubuntu Linux Operating System stealing devices from the bus, only 1 host or 1 guest can access devices at once, if they attempt to together you will crash both OS and your computer will lock)


Had to experiment around a bit. radeon.modeset=0 and amdgpu.runpm=0 are apparently effective ways to disable the Radeon Catalyst drivers Ubuntu automatically installed when you installed it (at least if the rx580 were plugged in then). If they weren’t, there is the small potential the radeon linux default settings aren’t necessary.

The things that are definitely necessary is passing to pci-stub (or VTIO as well as :
intel_iommu=on which tells to the system to load VT-d (amd_iommu=on for AMD CPUs)

I’ve done that by defining a pci-stub.ids=1002:67df,1002:aaf0

Ignore the last ID in my config, sorry if its confusing. 8086:a170. Stupid me thought I needed to use pci stub for USB audio for some reason.  You don’t.


Not the best diagram illustrating the setup, but you get the idea

The reason why there are two PCI stub ID’s is simple. The pci output bus for each discrete radeon rx580 graphics card actually has 2 devices! One audio device and one video device. They work together to output to the hdmi port MUX. (they mix the shit and throw it out hdmi). Your computer does all this stuff transparently normally, but to get the passthrough to work, because of the nature of the architecture, to use the VGA card you actually need to assign both sub-busses of the device, if that makes sense.

About MMIO Groups (Memory Mapped IO)

Hence why we have them inside the group, it’s called an MMIO group, essentially for the device to be passed, all sub-id’s need to be a part of it. This is actually the exact reason why 2 x rx580 setups don’t work, or are not regarded popularly to be workable (due to laziness/stupidity/inability probly), since the MMIO group of the devices is usually determined by the defining ID, so how can you group a vga device and an audio card in that group using the vendorID, if there are two cards on two pcie slots, with the same vendor ID! Anyways, you get the idea. To understand how this shiz work you really need to get a fundamental understanding of why some combinations can’t work, or why people think they might not, and then look to overcoming the problem. I was able to overcome this problem, so keep reading if you are trying to as well, its a royal pain in the butt, and I hope my article can help you along. Hopefully it’s as complete as it needs to be for you to complete Crossfire rx580 passthrough, but if it isn’t drop a message and I’ll try help you get it set.


There’s a lot of technology at work making the regular GPU on your windows OS go. You see that when configuring livirtd, that these addresses really really matter and essentially influence whether your PCIE passthrough setup is a muddled success, failure, or complete failure.

For most of us muddled success is quite acceptable. Providing we understand this isn’t for use in production. This is my home setup.

Step 7B: Ensure Linux Kernel Modules (ubuntu’s equivalent of windows Drivers) are loaded

Also ensuring to check your GRUB_CMDLINE_LINUX_DEFAULT is saved too (as done earlier):


Apologies for the extra text there ,8086:a170. You dont need the bit that says ‘,8086:a170’, it’s relevant but shouldn’t hurt though if your confused and just want it working 🙂

Step 7C: Making Linux use your changes, Sorting BOOT & INIT

Now we need to make sure that the GRUB CMDLINE default is actually being used, to do that we need to rebuild grub bootloader, so that the command gets included in the /boot/grub/grub.cfg or similar file. (basically boot options like windows safe mode but much more awwwsome). There is something important to know here GRUB CAN BREAK YOUR WINDOWS BOOT IF YOU HAVE DUAL BOOT. It caused me much aggrievance so THINK BEFORE ACTING.

In my case I had 3 disks. /dev/sda, /dev/sdb and /dev/sdc. Where /dev/sdb1 was the partition and /dev/sdb was the physical device that disk partition was stored. /dev/sda was my Windows 10 disk. Windows 10 has a MBR BIOS type bootloader installed to the disk on /dev/sda, if I instruct grub to update the bootloader, it will update all of them , dev/sda, dev/sdb and /dev/sdc (the bootloader isn’t stored on a partition, its stored on the first blocks of the physical device, hence /sdb /sdc and not sdb1.).

So whilst this shouldn’t cause DATA LOSS itself, it will cause bootloader loss if you run grub-update unaccordingly. It happened to me a few times, because I’m so noobish with windows bootloaders. So, the way to avoid breaking your windows MBR BIOS bootloader is simply to issue the grub-update command the device you wish to regenerate the bootloader for (as afterall the only reason we are doing this is to include that default command to make sure our pci stubs get passed thru early and drivers dont load up on the HOST OS which will kill the guest OS if they both have connections to those PCIE busses).

Simply then to apply the grub bootloader updater only to my Ubuntu /dev/sdb partition Linux runs only.

update-grub /dev/sdb

You may (most definitely will need to) to also update the initramfs , mainly because initramfs interacts with the module blacklist.

If you haven’t used modprobe.d before, it just disables your kernel drivers. Here is the 3 lines I added to the end of my file /etc/modprobe.d/blacklist.conf (make the file if it doesnt exist, and ensure .conf is the file extension or its ignored).


Might not be necessary, may be slightly different drivers for you to disable, etc. use lspci -v to find out ful details about what drivers your pcie cards are using, and you’ll be able to blacklist the correct driver.

In my case the reason why I blacklisted snd_hda_intel was because of this output from my lspci -v for that device revealed that kernel module driver used;

In my case it says kernel-driver in use:pci-stub because I’m writing this tutorial and have already configured it. Note that’s what it’ll say for your graphics card too once it is passed to pci-stub. And then, once thats done, and only then you’ll be able to use the card on the virtual machine, whew!!! I’m tired, I don’t know about you!

Just to give you an idea about the MMIO groups used by the VTIO/VT-d technologies, this is the structure of the tree of devices;

You can see our graphics cards at 01.0 and the MMIO [01] group consisting of the two sub chips on that bus 00.0, and 00.1.

We still need to make sure that pci-stub module is loaded. If it’s not in there already add pci_stub (note _ not -) in the configuration. Yes I put both in my configuration because it doesn’t matter, and I had a hell of a time getting this to work in Ubuntu kk).

Once done simply finally update the initramfs-tools, to ensure this change to /etc/initramfs-tools/modules gets compiled into the linux boot image. The Linux boot image is used at bootime, and contains included module drivers for Linux Kernel. Don’t miss this bit.

REBOOT!

Step 8: Checking it all worked right


This is what you should see. You want to see those pci-stub add’s against the device vendor ID’s. Don’t tell me how I did it, but as you can see both vendor ID’s get added, but not both MMIO groups. Yet, you will still find, that this works with multiple RX580 with same vendor ID (or whatever gpu you have). And I’m wondering if that has simply been fixed in the latest 16.04 kernel

For anyone having difficulty matching my environment;

root@adam:/home/adam# uname -a
Linux adam 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Also, more is to come including the libvirtd installation and configuration. To get it working from here, just install libvirtd as normal, and add the PCI devices that now become available with the 440FX and Q35 chipsets.

HAVE FUN y’all! I did spend a little while getting my head around this and hope that others can benefit or make comments of anything I’ve missed, or mistaken. It’s a huge piece of turning wheels!

Renaming a folder en-mass

I was figuring out a way to rename a folder en-mass which took me a few seconds, instead of about an hour of clicking.

Linux automation is a win.

# mmv "Documentary - National Geometry series-*.avi" "#1.avi"

I had some old tv series but they were all prefixed with the same filename ‘Documentary – National Geometry series’ before their volume number and title. So I found the above command to strip that out for all 100 of them, in a single command. Basically #1 is where * is in the source command regular expression, so everything after the source expression and between the *.

It’s a really nice utility and preferred on BSD systems to perl rename imo

Rapid Troubleshooting mail not arriving at its destination

Today I had a customer whose mail was not arriving at it’s destination. IN my customers case, they knew that it was arriving at the destination but going into the spam folder. This is likely due to blacklisting, however some ISP or clients dont know it’s reaching the spam folder at the other end, and since the most common cause of this happening is due to a missing SPF (Sending policy framework record), MX record or DKIM record it is possibly to rapidly check the DNS of each using dig, if the sender domain is known.

To check for the IP Blacklistings on a mailserver use it’s ip in one of the many spam a checker;

https://mxtoolbox.com/SuperTool.aspx?action=blacklist

In other cases it might be being caused by email bounces, due to the PTR, MX or DKIM records, and not even getting into the inbox, you can see that on the sending mail server using a simple grep command;

cat /var/log/maillog | grep -i status=bounced

You probably want to save the file as well

cat /var/log/maillog | grep -i status=bounced > bouncedemail.txt 

If you wanted to know which domains bounced email, if you’ve ensured all sending domains are correctly configured via DNS..

[root@api ~]# cat /var/log/maillog | grep -i status=bounced | awk '{print $7}'
to=<pi.magnum@generationgame.com>,
to=<john.partridge@boeeb.com>,

You could use sed to extract which domains failed..

Then you could use whois against the domains to reach out to the email contact with some automation that explains that ‘weve checked DKIM, MX and SPF and all are configured correctly and believe this is an error on your behalf, etc, blah blah’..

Such a thing would be a good idea to implement for some large email providers, and I’m sure you could automate the DNS checking as well. You could likely automate the whole thing, just by watching the logs and DNS records, and some intelligent grep and awking.

Not bad.

Lysncd fails with Error: Terminating since out of inotify watches.

This is a remarkably easy problem to solve.

From man:

The inotify API provides a mechanism for monitoring filesystem
       events.  Inotify can be used to monitor individual files, or to
       monitor directories.  When a directory is monitored, inotify will
       return events for the directory itself, and for files inside the
       directory.

The problem is a kernel based value, that determines how much memory is used by processes such as lsyncd. Lets just double check what value it is set to presently:

Check Inotify max_user_watches value presently

[root@server php]# cat /proc/sys/fs/inotify/max_user_watches
100000

Check the amount of available memory on the server to ensure it can deal with the increase

[root@server-new php]# free -m
             total       used       free     shared    buffers     cached
Mem:          7358       6577        781          1        376       2427
-/+ buffers/cache:       3773       3584
Swap:            0          0          0

The cache can be added to free, as a maximum saturation. Linux uses lots of memory it doesn’t really need for efficiency.

Each used inotify watch uses 1 kB RAM in 64 bit systems, for 32bit its half.

Increase Inotify watches to a higher value

sysctl fs.inotify.max_user_watches=200000

Magento Rewrite Woes … really woes

I had a customer this week that had some terrible rewrite woes with their magento site. They knew that a whole ton of their images were getting 404’s most likely because rewrite wasn’t getting to the correct filesystem path that the file resided. This was due to their cache being broken, and their second developer not creating proper rewrite rule.

As a sysadmin our job is not a development role, we are a support role, and in order to enable the developer to fix the problem, the developer needs to be able to see exactly what it is, enter the sysads task. I wrote this really ghetto script, which essentially hunts in the nginx error log for requests that failed with no such file, and then qualifies them by grepping for jpg file types. This is not a perfect way of doing it, however, it is really effective at identifying the broken links.

Then I have a seperate routine that strips the each of the file uri’s down to the filename, and locates the file on the filesystem, and matches the filename on the filesystem that the rewrite should be going to, as well as the incorrect path that the rewrite is presently putting the url to. See the script below:

#!/bin/bash

# Author: Adam Bull
# Company: Rackspace LTD, Hayes
# Purpose:
#          This customer has a difficulty with nginx rewriting to the incorrect file
# this script goes into the nginx error.log and finds all the images which have the broken rewrite rules
# then after it has identified the broken rewrite rule files, it searches for the correct file on the filesystem
# then correlates it with the necessary rewrite rule that is required
# this could potentially be used for in-place upgrade by developers
# to ensure that website has proper redirects in the case of bugs with the ones which exist.

# This script will effectively find all 404's and give necessary information for forming a rewrite rule, i.e. the request url, from nginx error.log vs the actual filesystem location
# on the hard disk that the request needs to go to, but is not being rewritten to file path correctly already

# that way this data could be used to create rewrite rules programmatically, potentially
# This is a work in progress


# These are used for display output
cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print "URL Request:",$21,"\nFilesystem destination missing:",$7"\n"}'
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print "URL Request:",$21,"\nFilesystem destination detected missing:",$7"\n"}'

# These below are used for variable population for locating actual file paths of missing files needed to determine the proper rewrite path destination (which is missing)
# we qualify this with only *.jpg files

cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print $7}' | sed 's/\"//g' |  sed 's/.*\///' | grep jpg > lost.txt
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print $7}' | sed 's/\"//g' |  sed 's/.*\///' | grep jpg >> lost.txt

# FULL REQUEST URL NEEDED AS WELL
cat /var/log/nginx/error.log /var/log/nginx/error.log.1 | grep 'No such file' | awk '{print "http://mycustomerswebsite.com",$21}' | sed 's/\"//g' | grep jpg > lostfullurl.txt
zcat /var/log/nginx/*error*.gz  | grep 'No such file' | awk '{print "http://customerwebsite.com/",$21}' | sed 's/\"//g' | grep jpg >> lostfullurl.txt

# The below section is used for finding the lost files on filesystem and pairing them together in variable pairs
# for programmatic usage for rewrite rules


while true
do
  read -r f1 <&3 || break
  read -r f2 <&4 || break
  printf '\n\n'
  printf 'Found a broken link getting a 404 at : %s\n'
  printf "$f1\n"
  printf 'Locating the correct link of the file on the filesystem: %s\n'
        find /var/www/magento | grep $f2
done 3<lostfullurl.txt 4<lost.txt

I was particularly proud of the last section which uses a ‘dual loop for two input files’ in a single while statement, allowing me to achieve the descriptions above.

Output is in the form of:


Found a broken link getting a 404 at :
http://customerswebsite.com/media/catalog/product/cache/1/image/800x700/9df78eab33525d08d6e5fb8d27136e95/b/o/image-magick-file-red.jpg
Locating the correct link of the file on the filesystem:
/var/www/magento/media/catalog/product/b/o/image-magick-file-red.jpg

As you can see the path is different on the filesystem to the url that the rewrite is putting the request to, hence the 404 this customer is getting.

This could be a really useful script, and, I see no reason why the script could not generate the rewrite rules programatically from the 404 failures it finds, it could actually create rules that are necessary to fix the problem. Now, this is not an ideal fix, however the script will allow you to have an overview either to fix this properly as a developer, or as a sysadmin to patch up with new rewrite rules.

I’m really proud of this one, even though not everyone may see a use for it. There really really is, and this customer is stoked, think of it like this, how can a developer fix it if he doesn’t have a clear idea of the things that are broken, and this is the sysads job,

Cheers &
Best wishes,
Adam

Recovering Corrupt RPM DB

I had a support specialist today that had an open yum task they couldn’t kill gracefully, after kill -9, this was happening

	
[root@mybox home]# yum list | grep -i xml
rpmdb: Thread/process 31902/140347322918656 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv-&gt;failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:
Error: rpmdb open failed
[root@mybox home]#

The solution to fix this is to manually erase the yumdb and to manually rebuild it;


[root@mybox home]# rm -f /var/lib/rpm/__*
[root@mybox home]# rpm --rebuilddb

Finding all of the php.ini Files and finding what the memory_limit in them is

Simple as it says.

root@aea-php-slave:~# grep -rnw '/etc' -e 'memory_limit'
/etc/php/5.6/cli/php.ini:393:memory_limit = -1
/etc/php/5.6/apache2/php.ini:393:memory_limit = 128M
/etc/php5/fpm/php.ini:406:memory_limit = 128M
/etc/php5/fpm/pool.d/www.conf.disabled:392:;php_admin_value[memory_limit] = 32M
/etc/php5/fpm/pool.d/site.conf:392:;php_admin_value[memory_limit] = 32M
/etc/php5/cli/php.ini:406:memory_limit = -1
/etc/php5/apache2/php.ini:406:memory_limit = 128M

p