Creating a Generative Adversarial Network and Visions of the Future

Featured

The people generated by the ASIC device do not exist. They are synthetic creations of the Generative Adversarial Network.

Over the last few years many will know I’ve been engaged in researches into blockchain. Particularly Ravencoin X16R,X16RV2 and KAWPOW, as well as the many blockchain explorers/trackers/scanners that I have written.

Recently I’ve become a little bit obsessed with GAN, a recently invented class of machine learning frameworks designed by Ian Goodfellow and his colleagues in 2014 and his colleagues at Nvidia.[1] Two neural networks contest with each other in a game (in the form of a zero-sum game, where one agent’s gain is another agent’s loss).

Stylegan 1024px early model with –size 256 parameter

Naturally, using modern GPU power that has traditionally been used for physics simulations, gaming, computational problems and things like cryptocurrency mining, it was a novel idea to consider that such technology could be used to create new novel data. Or be a fundamental resource of great power and adversarial adaptation in the playing of the game.

Trent Klein on Twitter: ""I can't believe it. The computer beaten by flesh  & blood." ~Doctor Pulaski (Peak Performance) #StarTrek #Strategema… "
Naturally, the machine lacks many inherent qualities of a human grandmaster knowledgeable in practice and study, but under the right circumstances or rule-set the machine in this case “data” is able to utilize the advantage of great speed, raw compute power, stamina and perfect reproduction of a strategy to beat the grandmaster at “Strategema”. This surely is then practice over study. This shows that machines with the right knowledge have an advantage. And that human beings with the right machines, have great power with their novel advantage to direct them.

I was very impressed at deepfakes, and the speed that deeplearning, machine learning and other technology has grown, and although I was generally disinterested by early A.I, such as “ALICE” and other polymorphic approaches to computer programming, I was particularly captivated by the deeplearning of Nvidia’s GAN. It appeared that through the correct processing of image boundaries of a very huge amount of “data”, a neural network, much like Data’s in startrek, really was capable of producing extremely novel applications in science and technology. For example, there is no reason why a similar approach could not be used to improve designs, or even build an entire product from start to finish without any human intervention. It certainly would seem then the cosmic idea of a “universal constructor”, first introduced in the popular game “Deus Ex”, is not such a strange idea. Certainly not when it is possible to apply the same methodology for face mixing and latent tracing as with Nvidia GAN, to chemical structures. Theoretically a machine that conceive an indefinite number of combinations, but can also discriminatingly qualify them in a similar way to a human being. An impressive feat.

Stylegan 1024px early model with –size 512 parameter.
The “main” sample.png file generated by the GAN modeler algorithm. These above images are used for “MIXING” (see below)
The power of the Adversarial Network is able to in the early stages produce very basic images that do not yet exceed human modeling and perceptional awareness (such as whether the image is real or not)

Predicting the Beginning and the End

To those that worry, about the technology for the future that will destroy human ingenuity and practicality – I think that the transformative power, and capability of GAN and technology like it, should allow us to create self improving machines that soon will become our guardians of the earth and the extended galaxy. A far fetched idea to some, but this technology makes it seem inevitable to me.

Stylegan 1024px latest model with –size 1024 parameter.
The trained GAN Model although has a few issues in some of the images, is nearly flawless in it’s production

It may not be very soon, but from what I can see already and imagine, the possibilities for this technology are truly endless, and it will likely be used, it may very well be used for exploring the universe from home. This technology is so simple at present, that the more complex forms of it’s application, theoretically could create entire universes, and with the sufficient compute and energy, it might be possible to discover many things about our universe without actually studying them. Simply provide a few simple rules, and the rest can be generated. Theoretically, anyway. Perhaps, then, we might be nearing a real explanation for the Hawkings Paradox, perhaps some thermal dynamic problems such as the total energy available at the beginning of the universe can be solved in a similar way through GAN type neural compute from data presently at the “middle” and latent images of stars very distant in the “beginning” or past (it takes a long time for light to reach the planet earth so most cosmic light is ancient). Using this data a new type of fundamental GAN that doesn’t just shape engineering, and novel artistic insight or design, or some chemistry simulation, but it may indeed allow us to predict nearly all things, and create a new type of computer system that is quite different from the one we are familiar.

When we mix the trained GAN Model generations we get new sets of variations;

Stylegan 1024px early model with –size 1024 parameter. This image particularly shows well the adapative nature of the Generative Adversarial Network and shows how the deep learning algorithm “learns” effectively faces and can “mix” any attributes using it’s learned data from it’s previous deeplearning training. to me It is very impressive.

A new Computer System

This new computer system would, theoretically, make efficiencies everywhere where we do not. Such as the adequate and measurable metric or data storage, redundancy, and things like satellite imagery and weather reporting. The neutral net device should theoretically be linkable to human consciousness, and to a greater system and create a new type of VR highway, that I predict will one day exist, optimizing many frequent challenges of modern society, that, until about 30-40 years ago, did not exist, until the abundance of data came along. GAN is a result of the abundance of data, but perhaps certain fundamental societal and technological evolutions in civilisation. Technology like GAN and blockchain might just be an inevitable byproduct or endproduct, of more data than we can humanly handle. And finding a way to use the data we have more efficiently, and to track it properly with automation, (such as with cloud compute), this is key. Really – the secret mystical understanding of the future of technology – was based on the understanding of the derivation of technology, society, and art, and the manner in which humanity interacts with that over a period of time. This reveals how science and art, and the society that practices it must change, rather than that the change applies to society, the society very much applies to the change.

Creating the Neural Network on Nvidia/CUDA

Creating the network is simple enough to do, and this can be done without a Docker Container on what I’d recommend would be an Ubuntu 20.04 LTS system. You can also use a docker container, however a Ubuntu 20.04 LTS system with the reference Nvidia drivers and a venv environment should be sufficient for our needs. It’s worth noting that if you intend to use the latest version of torch, python 3.8 is incompatible with torch v2, and I had some difficulty installing v1 on my linux system, simply because I was running python 3.8. It should work OK if you have a venv with python 3.7 or similar. Because this configuration can break a lot of things. It is highly recommended to use either Docker or venv, or both or either to achieve this.

Installing and Preparing the Datasets

#install venv
sudo apt-get install python3-pip

# do not do this as root, create a user for it [or use your regular user]
adduser someuser
virtualenv venv -p python3

# active venv (must be done where venv created)
source venv/bin/activate

# clone my repo
git clone https://github.com/ravenlandpush/sbgan

# cd to repo and download the celeba dataset 
# (http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html )
cd sbgan
python helper.py

# Prepare the JPEG data (--out datasetout pathtosearchsubdirsforimages)
python prepare_data.py --out data .


# install the necessary dependencies (note that torchvision 2 should be OK) I use it OK for this and you can skip requiring 1.x for this example
pip install torch pillow tqdm torchvision lmdb

# CUDA package names for thoroughness shouldnt be necessary if you've 3rd party nvidia drivers installed by Ubuntu 20.04 LTS.

# DANGEROUS step if you don't know what your doing
# apt-get install nvidia-cuda-dev nvidia-cuda-toolkit nvidia-cuda-toolkit-gcc 

# Start Training the Deeplearning Generative Adversarial Network with your dataset
python3 train.py --mixing /home/adam/GAN/sbgan/data

Congratulations! You’ve reached this far and your GAN is now training. You’ll notice though that it’s running probably quite slowly. For really decent performance you’ll want to have a number of GPU. I’d recommend running on Amazon were it not so expensive. You can get multiple GPU systems though between $8 and $15 an hour, so, relatively that’s not bad considering Tesla P100 gpu’s can set you back thousands a piece. For those that mean business, and for the many that work on GAN more full time they seem to be using DGX-1 which have 6 or 9 GPU builtin and are very small. Unfortunately they cost about $129,000. Although it’s still quite a specialist field, it reminds me of where bigdata was 15-20 years ago. The same could be said for enterprise linux.

Things do change. The last steps now after many weeks would be to run against the models that your generating.

A sample is saved every 100 iterations in the sample directory.

Generating from Modelling

Once your GAN has been “trained”, it should be possible to generate some really amazing mixers of images and I was taken aback by how effective some of the software of modernity has become at identifying things, even when the software does not know what it is, detecting the boundary and “putting things in the right place” is all that matters to us.. very cool.

# Generate from the Trained Models created in checkpoint folder (happens as training goes by)
# Use size 8,16,32,64,128,256,512,1024, etc
# depending how far along the training

python3 generate.py /home/adam/GAN/sbgan/checkpoint/train_step-4.model --size 64 --n_row 8 --n_col 8

python3 generate.py /home/adam/GAN/sbgan/checkpoint/train_step-4.model --size 64 --n_row 8 --n_col 8

The Checkpoints for the GAN are generated in ./checkpoints, this allow you to retrain from any specific point and to compare or merge certain image sets later on if you wish to experiment with greater complexity

The Final results

I really love GAN’s now πŸ˜€

Just for fun I wrote this script that can automatically pull data in and out of a docker container.

#!/bin/bash
# this script indefinitely makes a new face every x moments
# deep fake y'all nvidia cuda stylee

length=1000000
for (( i = 0; i <= $length; i=i+4 )) ; do
j=$(($i + 1))
k=$(($j + 1))
l=$(($k + 1))
echo "Processing face $i,$j,$k,$l";

docker run --gpus all -it --rm -v `pwd`:/scratch --user $(id -u):$(id -g) stylegan2ada:latest bash -c     "(cd /scratch && DNNLIB_CACHE_DIR=/scratch/.cache python3 generate.py --trunc=1 --seeds=$i,$j,$k,$l --outdir=out --network=https://nvlabs-fi-cdn.nvidia.com/stylegan2-ada/pretrained/metfaces.pkl)" > /dev/null

sleep 10


done
             

Simple but, cool. meh. As you can see this one uses the stylegan2-ada pretained metfaces pkl model from nvlabs. Not bad for a quick poke around at a new subject.

Calculating the Average Hits per minute en-mass for thousands of sites

So, I had a customer having some major MySQL woes, and I wanted to know whether the MySQL issues were query related, as in due to the frequency of queries alone, or the size of the database. VS it being caused by the number of visitors coming into apache, therefore causing more frequency of MySQL hits, and explaining the higher CPU usage.

The best way to achieve this is to inspect /var/log/httpd with ls -al,

First we take a sample of all of the requests coming into apache2, as in all of them.. provided the customer has used proper naming conventions this isn’t a nightmare. Apache is designed to make this easy for you by the way it is setup by default, hurrah!

[[email protected] logparser]# time tail -f /var/log/httpd/*access_log > allhitsnow
^C

real	0m44.560s
user	0m0.006s
sys	0m0.031s

Time command prefixed here, will tell you how long you ran it for.

[[email protected] logparser]# cat allhitsnow | wc -l
1590

The above command shows you the number of lines in allhitsnow file, which was written to with all the new requests coming into sites from all the site log files. Simples! 1590 queries a minute is quite a lot.

Site keeps on going down because of spiders

So a Rackspace customer was consistently having an issue with their site going down, even after the number of workers were increased. It looked like in this customers case they were being hit really hard by yahoo slurp, google bot, a href bot, and many many others.

So I checked the hour the customer was affected, and found that over that hour just yahoo slurp and google bot accounted for 415 of the requests. This made up like 25% of all the requests to the site so it was certainly a possibility the max workers were being reached due to spikes in traffic from bots, in parallel with potential spikes in usual visitors.

[[email protected] logs]#  grep '01/Mar/2017:10:' access_log | egrep -i 'www.google.com/bot.html|http://help.yahoo.com/help/us/ysearch/slurp' |  wc -l
415

It wasn’t a complete theory, but was the best with all the available information I had, since everything else had been checked. The only thing that remains is the number of retransmits for that machine. All in all it was a victory, and this was so awesome, I’m now thinking of making a tool that will do this in more automated way.

I don’t know if this is the best way to find google bot and yahoo bot spiders, but it seems like a good method to start.

Count number of IP’s over a given time period in Apache Log

So, a customer had an outage, and wasn’t sure what caused it. It looked like some IP’s were hammering the site, so I wrote this quite one liner just to sort the IP’s numerically, so that uniq -c can count the duplicate requests, this way we can count exactly how many times a given IP makes a request in any given minute or hour:

Any given minute

# grep '24/Feb/2017:10:03' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any given hour

# grep '24/Feb/2017:10:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given day

# grep '24/Feb/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given Month

# grep '/Feb/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any Given Year

# grep '/2017:' /var/www/html/website.com/access.log | awk '{print $1}' | sort -k2nr | uniq -c

Any given year might cause dupes though, and I’m sure there is a better way of doing that which is more specific

How to limit the amount of memory httpd is using on CentOS 7 with Cgroups

CentOS 7, introduced something called CGroups, or control groups which has been in the stable kernel since about 2006. The systemD unit is made of several parts, systemD unit resource controllers can be used to limit memory usage of a service, such as httpd control group in systemD.

# Set Memory Limits for a systemD unit
systemctl set-property httpd MemoryLimit=500MB

# Get Limits for a systemD Unit
systemctl show -p CPUShares 
systemctl show -p MemoryLimit

Please note that OS level support is not generally provided with managed infrastructure service level, however I wanted to help where I could hear, and it shouldn’t be that difficult because the new stuff introduced in SystemD and CGroups is much more powerful and convenient than using ulimit or similar.

Using Sar to tell a story

So, a customer is experiencing slowness/sluggishness in their app. You know there is not issue with the hypervisor from instinct, but instinct isn’t enough. Using tools like xentop, sar, bwm-ng are critical parts of live and historical troubleshooting.

Sar can tell you a story, if you can ask the storyteller the write questions, or even better, pick up the book and read it properly. You’ll understand what the plot, scenario, situation and exactly how to proceed with troubleshooting by paying attention to these data and knowing which things to check under certain circumstances.

This article doesn’t go in depth to that, but it gives you a good reference of a variety of tests, the most important being, cpu usage, io usage, network usage, and load averages.

CPU Usage of all processors

# Grab details live
sar -u 1 3

# Use historical binary sar file
# sa10 means '10th day' of current month.
sar -u -f /var/log/sa/sa10 

CPU Usage of a particular Processor

sar -P ALL 1 1

‘-P 1’ means check only the 2nd Core. (Core numbers start from 0).

sar -P 1 1 5

The above command displays real time CPU usage for core number 1, every 1 second for 5 times.

Observing Changes in Memory over time

sar -r 1 3

The above command provides memory stats every 1 second for a total of 3 times.

Observing Swap usage over time

sar -S 1 5

The above command reports swap statistics every 1 seconds, a total 3 times.

Overall I/O activity

sar -b 1 3 

The above command checks every 1 seconds, 3 times.

Individual Block Device I/O Activities

This is a useful check for LUN , block devices and other specific mounts

sar -d 1 1 
sar -p d

DEV – indicates block device, i.e. sda, sda1, sdb1 etc.

Total Number processors created a second / Context switches

sar -w 1 3

Run Queue and Load Average

sar -q 1 3 

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes. β€œ1 3” reports for every 1 seconds a total of 3 times.

Report Network Statistics

sar -n KEYWORD

KEYWORDS Available;

DEV – Displays network devices vital statistics for eth0, eth1, etc.,
EDEV – Display network device failure statistics
NFS – Displays NFS client activities
NFSD – Displays NFS server activities
SOCK – Displays sockets in use for IPv4
IP – Displays IPv4 network traffic
EIP – Displays IPv4 network errors
ICMP – Displays ICMPv4 network traffic
EICMP – Displays ICMPv4 network errors
TCP – Displays TCPv4 network traffic
ETCP – Displays TCPv4 network errors
UDP – Displays UDPv4 network traffic
SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
ALL – This displays all of the above information. The output will be very long.

sar -n DEV 1 1

Specify Start Time

sar -q -f /var/log/sa/sa11 -s 11:00:00
sar -q -f /var/log/sa/sa11 -s 11:00:00 | head -n 10

Grabbing network activity from server without network utility

So, is it possible to look at a network interfaces activity without bwm-ng, iptraf, or other tools? Yes.

while true do
RX1=`cat /sys/class/net/${INTERFACE}/statistics/rx_bytes`
TX1=`cat /sys/class/net/${INTERFACE}/statistics/tx_bytes`
DOWN=$(($RX1-$RX2))
UP=$(($TX1-$TX2))
DOWN_Bits=$(($DOWN * 8 ))
UP_Bits=$(($UP * 8 ))
DOWNmbps=$(( $DOWN_Bits >> 20 ))
UPmbps=$(($UP_Bits >> 20 ))
echo -e "RX:${DOWN}\tTX:${UP} B/s | RX:${DOWNmbps}\tTX:${UPmbps} Mb/s"
RX2=$RX1; TX2=$TX1
sleep 1; done

I found this little gem yesterday, but couldn’t understand why they had not used clear. I guess they wanted to log activity or something… still this was a really nice find. I can’t remember where I found it yesterday but googling part of it should lead you to the original source πŸ˜€

Checking requests to apache2 webserver during downtime

A customer of ours was having some serious disruptions to his webserver, with 15 minute outages happening here and there. He said he couldn’t see an increase in traffic and therefore didn’t understand why it reached maxclients. Here was a quick way to prove whether traffic really increased or not by directly grepping the access logs for the time and day in question and using wc -l to count them, and a for loop to step thru the minutes of the hour in between the events.

Proud of this simple one.. much simpler than a lot of other scripts that do the same thing I’ve seen out there!

[email protected]:/var/log/apache2# for i in `seq 01 60`;  do  printf "total visits: 13:$i\n\n"; grep "12/Jul/2016:13:$i" access.log | wc -l; done

total visits: 13:1

305
total visits: 13:2

474
total visits: 13:3

421
total visits: 13:4

411
total visits: 13:5

733
total visits: 13:6

0
total visits: 13:7

0
total visits: 13:8

0
total visits: 13:9

0
total visits: 13:10

30
total visits: 13:11

36
total visits: 13:12

30
total visits: 13:13

29
total visits: 13:14

28
total visits: 13:15

26
total visits: 13:16

26
total visits: 13:17

32
total visits: 13:18

37
total visits: 13:19

31
total visits: 13:20

42
total visits: 13:21

47
total visits: 13:22

65
total visits: 13:23

51
total visits: 13:24

57
total visits: 13:25

38
total visits: 13:26

40
total visits: 13:27

51
total visits: 13:28

51
total visits: 13:29

32
total visits: 13:30

56
total visits: 13:31

37
total visits: 13:32

36
total visits: 13:33

32
total visits: 13:34

36
total visits: 13:35

36
total visits: 13:36

39
total visits: 13:37

70
total visits: 13:38

52
total visits: 13:39

27
total visits: 13:40

38
total visits: 13:41

46
total visits: 13:42

46
total visits: 13:43

47
total visits: 13:44

39
total visits: 13:45

36
total visits: 13:46

39
total visits: 13:47

49
total visits: 13:48

41
total visits: 13:49

30
total visits: 13:50

57
total visits: 13:51

68
total visits: 13:52

99
total visits: 13:53

52
total visits: 13:54

92
total visits: 13:55

66
total visits: 13:56

75
total visits: 13:57

70
total visits: 13:58

87
total visits: 13:59

67
total visits: 13:60

[email protected]:/var/log/apache2# for i in `seq 01 60`; do printf “total visits: 12:$i\n\n”; grep “12/Jul/2016:12:$i” access.log | wc -l; done
total visits: 12:1

169
total visits: 12:2

248
total visits: 12:3

298
total visits: 12:4

200
total visits: 12:5

341
total visits: 12:6

0
total visits: 12:7

0
total visits: 12:8

0
total visits: 12:9

0
total visits: 12:10

13
total visits: 12:11

11
total visits: 12:12

30
total visits: 12:13

11
total visits: 12:14

11
total visits: 12:15

13
total visits: 12:16

16
total visits: 12:17

28
total visits: 12:18

26
total visits: 12:19

10
total visits: 12:20

19
total visits: 12:21

35
total visits: 12:22

12
total visits: 12:23

19
total visits: 12:24

28
total visits: 12:25

25
total visits: 12:26

30
total visits: 12:27

43
total visits: 12:28

13
total visits: 12:29

24
total visits: 12:30

39
total visits: 12:31

35
total visits: 12:32

25
total visits: 12:33

22
total visits: 12:34

33
total visits: 12:35

21
total visits: 12:36

31
total visits: 12:37

31
total visits: 12:38

22
total visits: 12:39

39
total visits: 12:40

11
total visits: 12:41

18
total visits: 12:42

11
total visits: 12:43

28
total visits: 12:44

19
total visits: 12:45

27
total visits: 12:46

18
total visits: 12:47

17
total visits: 12:48

22
total visits: 12:49

29
total visits: 12:50

22
total visits: 12:51

31
total visits: 12:52

44
total visits: 12:53

38
total visits: 12:54

38
total visits: 12:55

41
total visits: 12:56

38
total visits: 12:57

32
total visits: 12:58

26
total visits: 12:59

31
total visits: 12:60

Tuning Apache2 for high Traffic Spikes

So this came up recently where a customer was asking if we could tune their apache2 for higher traffic. The best way to do this is to benchmark the site to double the traffic expected, this should be a good measure of whether the site is going to hold up..

# Use Apachebench to test the local requests
ab -n 1000000 -c 1000 http://localhost:80/__*index.html

Benchmarking localhost (be patient)
Completed 100000 requests
Completed 200000 requests
Completed 300000 requests
Completed 400000 requests
Completed 500000 requests
Completed 600000 requests
Completed 700000 requests
Completed 800000 requests
Completed 900000 requests
Completed 1000000 requests
Finished 1000000 requests

Server Software:        Apache/2.2.15
Server Hostname:        localhost
Server Port:            80

Document Path:          /__*index.html
Document Length:        5758 bytes

Concurrency Level:      1000
Time taken for tests:   377.636 seconds
Complete requests:      1000000
Failed requests:        115
   (Connect: 0, Receive: 0, Length: 115, Exceptions: 0)
Write errors:           0
Total transferred:      6028336810 bytes
HTML transferred:       5757366620 bytes
Requests per second:    2648.05 [#/sec] (mean)
Time per request:       377.636 [ms] (mean)
Time per request:       0.378 [ms] (mean, across all concurrent requests)
Transfer rate:          15589.21 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   52  243.0     22   15036
Processing:     0  282 1898.4     27   81404
Waiting:        0  270 1780.1     24   81400
Total:          6  334 1923.7     50   82432

Percentage of the requests served within a certain time (ms)
  50%     50
  66%     57
  75%     63
  80%     67
  90%     84
  95%   1036
  98%   4773
  99%   7991
 100%  82432 (longest request)



# During the benchmark test you may wish to use sar to indicate general load and io
stdbuf -o0 paste <(sar -q 10 100) <(sar 10 100) | awk '{printf "%8s %2s %7s %7s %7s %8s %9s %8s %8s\n", $1,$2,$3,$4,$5,$11,$13,$14,$NF}'

# Make any relevant adjustments to httpd.conf threads

# diff /etc/httpd/conf/httpd.conf /home/backup/etc/httpd/conf/httpd.conf
103,108c103,108
< StartServers       2000
< MinSpareServers    500
< MaxSpareServers   900
< ServerLimit      2990
< MaxClients       2990
< MaxRequestsPerChild  20000
---
> StartServers       8
> MinSpareServers    5
> MaxSpareServers   20
> ServerLimit      256
> MaxClients       256
> MaxRequestsPerChild  4000
-----------------------------------

In this case we increased the number of startservers and minspareservers. Thanks to Jacob for this.