High-Resolution Multi-Scale Neural Texture Synthesis (Mert Toka Presentation)

Xavier Snelgrove paper: (paper)

Texture synthesis:

"Texture synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content." (source)

Preserves image features

Does not analyze semantically

Brief Explanation: https://www.youtube.com/watch?v=Uo6hFVRsjpA

Images:

(If you need to zoom in the image, you can do so by downloading it)

Arguments

Long Arg	Short Arg	Usage	Default	Short Desc.
--source	-s	-s SOURCE [SOURCE ... ]	None	List of file to use as source textures
--output-width --output-height	-ow, -oh	-ow NUMBER, -oh NUMBER	512, 512	Output width and height
--octaves	-o	-o NUMBER	4	Number of octaves
--layers	-l	-l LAYERS [LAYERS ... ]	[2,7]	Which layers to match gram matrices on
--join-mode	-j	-j {average, max}	average	How to combine gram matrices when multiple sources given
--mul		--mul NUMBER	1.0	Multiply target grams by this amount
--seed		--seed {random, symmetric}	random	How to seed the optimization
--source-scale	-ss	-ss NUMBER	None	How much to scale the source image by
--source-width	-sw	-sw NUMBER	None	Scale source to this width. Mutually exclusive with source-scale
--padding-mode	-p	-p {valid, same}	valid	What boundary condition to use for convolutions
--tol		--tol NUMBER	1e-09	Gradient scale at which to terminate optimization
--data-dir	-d	-d DIRECTORY	"model_data"	Where to find the VGG weight files
--output-dir		--output-dir DIRECTORY	"outputs"	Where to save the generated outputs
--max-iter	-m	-m NUMBER	500	Maximum iterations for the L-BFGS-B optimizer
--output-prefix	-op	-op PREFIX	"out"	Prefix to append to output directory
--save-every	-n	-n NUMBER	10	Save an in-progress optimization image every SAVE_EVERY iterations
--count	-c	-c NUMBER	1	How many images to generate simultaneously
--if-weight		--if-weight NUMBER	1.0	Inter-frame loss weight
--if-shift		--if-shift NUMBER	5.0	How many pixel-shift should inter-frame loss approximate?
--if-order		--if-order NUMBER	2.0	How many frames should we 'tie' together?

Details:

Figure from the paper that illustrates the network architecture

Tensorflow: Library that abstracts the details of common operations performed in machine learning

Keras: Python interface for Tensorflow

Gram matrices: Given a set of $\textbf{V}$ of $\textbf{m}$ vectors, the Gram matrix $\mathbf{G}$ is the matrix of all possible inner products of $\mathbf{V}$ . [ $g_{ij} = \mathbf{v_i}^T \mathbf{v_j}$ ]

holds correlations of feature maps stored in the hidden layers

powerful representation of the texture

synthesize new images whose Gram matrix is close to that of an example image via gradient descent

ImageNet: Image database that has an average of 500 images per label

VGG architectures: Pre-trained CNNs on ImageNet dataset. Keras now includes VGG-16 and VGG-19. VGG-19 network is 19 layers deep (16 convolutional + 3 fully connected) and can classify images into 1000 object categories.

Gaussian Pyramid: An image processing technique that iteratively blurs and down-samples the image.

Download & Install:

https://github.com/wxs/subjective-functions

Tested on Ubuntu 18.04 with Python 3.6.12.

possibly will run the same on MacOS

git clone https://github.com/wxs/subjective-functions.git
cd subjective-functions

pip3 install Pillow
pip3 install scipy==1.5.2
pip3 install numpy==1.19.2
pip3 install --user --upgrade tensorflow-gpu
pip3 install --user --upgrade tensorboard
pip3 install keras==2.3.1
pip3 install --user --upgrade tensorflow-gpu==1.14.0
pip3 install h5py==2.10.0

KERAS_BACKEND=tensorflow python3 synthesize.py -s bark.jpg


# If you would like to use different arguments, simply append them at 
# the end of the above line. e.g.:
KERAS_BACKEND=tensorflow python3 synthesize.py -s bark.jpg -o 1 -l 3 5 -mul 1.5

# Sequencing multiple runs back to back (put a semi column (;) in between different lines). e.g.:
KERAS_BACKEND=tensorflow python3 synthesize.py -s bark.jpg -l 3 8 -mul 1.5 ; KERAS_BACKEND=tensorflow python3 synthesize.py -s bark.jpg -o 2 -l 2 5 ;

Possible roadblock:

Modify h5py library in /home/<USERNAME>/.local/lib/python3.6/site-packages/keras/engine/saving.py

Line 273:

model_config = json.loads(model_config.decode('utf-8'))

model_config = json.loads(model_config)#.decode('utf-8'))

Line 278:

original_keras_version = model_weights_group['keras_version'].decode('utf8')

original_keras_version = model_weights_group['keras_version']#.decode('utf8')

Line 282:

original_backend = model_weights_group['backend'].decode('utf8')

original_backend = model_weights_group['backend']#.decode('utf8')