Yesterday I created a proof of concept script, which basically goes off and identifies the hitcounts of a website, and can give a technician within a short duration of time (minutes instead of hours) exactly where hitcounts are coming from and where.
This is kind of a tradeoff, between a script that is automated, and one that is flexible.
The end goal is to provide a hitcount vs memory commit metric value. A NEW TYPE OF METRIC! HURRAH! (This is needed by the industry IMO).
And also would be nice to generate graphing and mean, average, and ranges, etc. So can provide output like ‘stat’ tool. Here is how I have progress
#!/bin/bash # # Author: Adam Bull, Cirrus Infrastructure, Rackspace LTD # Date: March 20 2017 # Use: This script automates the analysis of webserver logs hitcounts and # provides a breakdown to indicate whether outages are caused by website visits # In correlation to memory and load avg figures # Settings # What logfile to get stats for logfile="/var/log/httpd/google.com-access.log" # What year month and day are we scanning for minute/hour hits year=2017 month=Mar day=9 echo "Total HITS: MARCH" grep "/$month/$year" "$logfile" | wc -l; # Hours for i in 0{1..9} {10..24}; do echo " > 9th March 2017, hits this $i hour"; grep "$day/$month/$year:$i" "$logfile" | wc -l; # break down the minutes in a nested visual way thats AWsome # Minutes for j in 0{1..9} {10..60}; do echo " >>hits at $i:$j"; grep "$day/$month/$year:$i:$j" "$logfile" | wc -l; done done
Thing is, after I wrote this, I wasn’t really happy, so I refactored it a bit more;
#!/bin/bash # # Author: Adam Bull, Cirrus Infrastructure, Rackspace LTD # Date: March 20 2017 # Use: This script automates the analysis of webserver logs hitcounts and # provides a breakdown to indicate whether outages are caused by website visits # In correlation to memory and load avg figures # Settings # What logfile to get stats for logfile="/var/log/httpd/someweb.biz-access.log" # What year month and day are we scanning for minute/hour hits year=2017 month=Mar day=9 echo "Total HITS: $month" grep "/$month/$year" "$logfile" | wc -l; # Hours for i in 0{1..9} {10..24}; do hitsperhour=$(grep "$day/$month/$year:$i" "$logfile" | wc -l;); echo " > $day $month $year, hits this $ith hour: $hitsperhour" # break down the minutes in a nested visual way thats AWsome # Minutes for j in 0{1..9} {10..59}; do hitsperminute=$(grep "$day/$month/$year:$i:$j" "$logfile" | wc -l); echo " >>hits at $i:$j $hitsperminute"; done done
Now it’s pretty leet.. well, simple. but functional. Here is what the output of the more nicely refined script; I’m really satisfied with the tabulation.
[root@822616-db1 automation]# ./list-visits.sh Total HITS: Mar 6019301 > 9 Mar 2017, hits this hour: 28793 >>hits at 01:01 416 >>hits at 01:02 380 >>hits at 01:03 417 >>hits at 01:04 408 >>hits at 01:05 385 ^C