2. Digiti Sonus - Advanced Interactive Fingerprint Sonification Using Visual Feature Analysis
Overview:
In this final project, I investigated how the fingerprints’ visual features could be extracted using image processing algorithms, and sonified. In order to expand the diversity of sound in my on-going interactive art installation “Digiti Sonus” I realized I should explore more detailed visual features of fingerprints, and I chose three visual features of fingerprints:
1) Minutiae, 2) Area of fingerprint and 3) Angle of fingerprint.
The most unique characteristics of fingerprints, called Minutiae, determine the pitch of sound over time in art installation. Using skeletonization and “Thinning” algorithm, I extracted Minutiae’s positions in a fingerprint. Area of fingerprint determines the overall fundamental Frequency range, and it was extracted by thresholding brightness of a fingerprint. Angle of fingerprint also has a key point to alter the timbre of sound. In FM synthesis, Fingerprint area maps to harmonicity ratio and angle maps to starting and ending point of one single envelope that controls modulation index. Push pressure maps intuitively to overall amplitude and duration, so that stronger imprints onto the fingerprint scanner result in stronger sounds.
1. Minutiae
Here is my initial process of skeletonization of few fingerprints using Matlab.
As you can see, after skeletonization process, there are points that have two/three neighbors, which could be possibly called as Minutiae. However, it’s not that super accurate because in second step of skeletonization, there were some data loss, and wrong line extraction that resulted in finding wrong minutiae. In order to get the accurate positions of Minutiae, I needed to use another method; I tried edge detection first, and moved on “Thinning” method.
Thinning
Reference: http://homepages.inf.ed.ac.uk/rbf/HIPR2/thin.htm
Thinning is a morphological operation that is used to remove selected foreground pixels from binary images, somewhat like erosion or opening. This procedure erodes away the boundaries of foreground objects as much as possible, but does not affect pixels at the ends of lines.
The thinning process must be performed multiple times until there is nothing left to thin.
Here are some thinning tests with different thresholds implemented in Processing.
(Raw image – Thresholding image – Thinning image – Minutiae)
Analysis: Overall, when I did test with several fingerprints, 0.2f threshold value seemed good for getting minutiae accurately.
2. Area of Fingerprint
In art installation, audience put their fingerprints in different position and area. It’s also interesting aspect of user intention, and I investigated the area of fingerprint using the simple thresholding process.
3. PCA (Principle Component Analysis)
Audience can apply their fingers to the fingerprint sensor at any angle. Usually, users touch the sensor near 0° (perfectly vertical orientation), however they often slightly rotate the finger, generally depending on the particular finger (thumb, index, etc.) and hand (left versus right). This analysis of fingerprint angle was based on PCA (Principal Component Analysis).
The input data are the x and y positions of the pixel above the brightness threshold. These data points are normalized by the mean of each dimension. Next, by computing eigenvalues and eigenvectors, the principal components are acquired. I regarded the first eigenvector (principal component) as the direction of input fingerprint image. Since principal components are always orthogonal and the data is two-dimensional, the second component gives no additional information.
4. Final 3D image / sonification
After all these visual feature extractions, I converted the 2D fingerprint image to 3D image, and applied the visual features on the 3D image as well.
Sonification
Sonification: When animation is going on, the positions of minutiae (= magnitude of frequency) were sent via UDP to MaxMSP. Now, there are 127 buffer, and it was mapped to 60~2000hz, but the frequency range is changed depending on the position and area of fingerprint. And as animation creates the ring shape, and touch the green colored minutiae, a sound in designated pitch is played and looped as time goes by. Overall, the positions of minutiae can be used as a part of musical score, and creates some short melody over time. Right now, only minutiae’s position is sent to MaxMSP, but not area and angle of fingerprint. I should calculate the number of red pixels, and position of fingerprint. And then I will send the value to Maxmsp to control both frequency range and harmonicity ratio of FM synthesis. And eigenvector value will be also sent to control the modulation index in MaxMSP. Those two are the things I should do in the near future.
Digiti Sonus @ EOYS 2013
Digiti Sonus was exhibited in End of Year Show 2013 in TransLAB.
Digiti Sonus @ NIME 2013 (Daejeon+Seoul)
Digiti Sonus was presented and exhibited at NIME 2013 in Daejeon+Seoul.
