Automated 3D Particle Field Extraction and Tracking System Using Digital in-line Holography

Digital holography for 3D particle field extraction and tracking is an active research topic. It has a great application in realizing characterization of micro-scale structures in microelectromechanical systems (MEMS) with high resolution and accuracy. In-line configuration is studied in this study as the fundamental structure of a digital holography system. Digital holographic approach, not only eliminates wet chemical processing and mechanical scanning, but also enables the use of complex amplitude information inaccessible by optical reconstruction, thereby allowing flexible reconstruction algorithms to achieve optimization of specific information. However, owing to the inherently low pixel resolution of solid-state imaging sensors, digital holography gives poor depth resolution for images. This problem severely impairs the usefulness of digital holography especially in densely populated particle fields. This study describes a system that significantly improves particle axial-location accuracy by exploring the reconstructed complex amplitude information, compared with other numerical reconstruction schemes that are merely traditional optical reconstruction. Theoretical analysis and experimental results demonstrate that in-line configuration presents advantageous in enhancing the system performance. Greater flexibility of the system, higher lateral resolution and lower speckle noise can be achieved.


INTRODUCTION
In conventional holography, holograms are photographically recorded and optically reconstructed. The applications of the in-line system are greatly restricted due to the interactive influence of the coaxial diffraction wave components during reconstruction [1] . It is off-axis holography that has been playing active roles in optical metrology as well as three-dimensional visualization. Same situation is in digital holography. Off-axis setup [2,3] is widely adopted in the digital holographic systems with many successful metrological applications.
To explore potential metrological applications of digital holography in particle field extraction, the small size of the microstructure and the requirement of high resolution as well as accuracy are the two major challenges. However, above drawbacks of the off-axis digital holography place difficulties to meet the demands. Instead, in-line setup is free from such limitations and matches well with the needs of realizing accurate characterizing to micro-scale objects. Although applications of in-line setup depend on effective solutions to its inherent problem, the unique properties of digital holography, such as the digital forms of the optical wavefields, together with the powerful numerical techniques, offer more flexibility to manipulate the information so as to achieve the goal. The reported phase-shifting digital holography [4] was a successful example based on in-line setup and applied in microscopy [5] .
The position and size of particles can be obtained from the numerically reconstructed particle field. Particle tracking can also be obtained by measuring particle displacement over two briefly spaced exposures.
The large depth of focus results in poor depth resolution and thus severely compromises the 3D capability of digital particle holography. One way to solve the depth resolution problem is to analyze the diffraction patterns in non-image planes instead [6] Several non-image plane methods have been introduced. Because these methods do not require the knowledge of the exact location of the in-focus planes, they are less sensitive to the large depth of focus of the reconstructed images. A common drawback of these methods, however, is that in the non-image planes the diffraction pattern of individual particles must be segmented first. A new approach, introduced in this study is to use the complex amplitude of the reconstruction wave that is uniquely available in digital particle holography to extract particles. Optical reconstruction of holograms does not provide convenient access to the complex amplitude of the reconstruction wave because sensors used for interrogating the reconstructed 3D image field only respond to intensity. In numerical reconstruction the complex amplitude is readily available. In this study, we have developed an automated system for extracting particles from holograms using the complex amplitude of the reconstruction wave. The results have shown that this method significantly improves the depth resolution in digital particle holography as compared with the intensity methods that are often used in traditional optical reconstruction.

IN-LINE DIGITAL HOLOGRAPHY SYSTEM
The theoretical background for numerical reconstruction of digital holography is the scalar diffraction theory. According to the different diffraction properties of the wave fields and system configurations, Fresnel transform diffraction formula and convolution type diffraction integral [7] have been developed for the numerical reconstruction of Fresnel holograms, while Fourier approach [8] was proposed for the case of lens less Fourier transform digital holography. With regard to the in-line Fresnel holographic system in our studies, Fresnel approach is used to reconstruct the image wavefield numerically. Suppose represents the intensity distribution of the in-line hologram resulted from the interference of the object wavefield O(x H ,y H ) and reference wavefield R(x H ,y H ) in the recording plane (x H ,y H ).
Digital recording mechanisms: Digital recording of holograms has to fulfill the requirement of sampling theorem if good quality reconstruction is expected in digital holography [1] . In order to ensure effective use of the CCD pixels, it is important that sampling theorem should be fulfilled across the whole CCD sensor. The parameters used in our experiments are: pixel number N x N = 1024 x 1024 and a pixel size ∆N = 9 µm, the maximum interference angle is limited to be α max =λ /2 ∆N = 1.7° for λ = 0.532 µm.
To an object of certain lateral size, the recording distance where it is placed away from the CCD target has to be greater than a minimum value D min , so that the spherical wavelet from each poin on the object interferes with the reference wave in the recording plane at an angle no greater than α max . Theoretical analysis of the minimum allowable recording distance in the cases of the in-line setup is carried out as follows.
For the in-line setup, the possible maximum interference angle, as illustrated in Fig. 1, occurs between the rays from the lowest edge point on the object and the normally incident plane reference wave at the topmost point of the CCD sensor.
Supposed that the centers of the object and CCD target are both located on the optical axis of the system, the following relationship regarding the minimum allowable distance D min:in-line can be obtained as: for small α max , where L Oy denotes the lateral size of the object in Y-axis and L CCD = N x∆N = 9.216 mm is the sensor size of the CCD used in the experiments. It is demonstrated that D min:in-line increases linearly with the object size at a slope of (1/2α max ).  [2,3] . Assuming that the real image wave is used in particle extraction, we first study its characteristic near the in-focus plane of a particle. Because Fresnel diffraction is a valid simplification of the Lorenz-Mie theory to predict forward light scattering [6] . We will use the Fresnel diffraction formula to introduce the wave characteristic. Let ' ' j A (x , y ) denote the amplitude transmittance function of particle P j , its scattered wave in the recording plane can be written as and z i is the axial distance between the particle and the recording plane.
In the hologram reconstruction, the real image wave of particle P j is obtained from the propagation of the conjugate of the scattered wave, o i * and it can be written as In the in-focus plane where z=z j , the convolution between two diffraction kernels in Eq. (5) results in a delta function and thereby r j = A j *. In the case of opaque particles the amplitude transmittance function A j (or A j *) can be modeled by a real function, therefore the imaginary part of r j vanishes in the in-focus plane of particle P j . This suggests the possibility of using the imaginary part of the image wave to determine the particle axial position.

DIPPING CHARACTERISTICS OF TOTAL RECONSTRUCTION FIELD
Ideally, the extraction of reconstructed particle P j should operate solely on its real image wave r j during hologram reconstruction. However, in reality we cannot separate the real image wave r j from the total reconstruction field U, which also consists of the virtual image wave v j of this particle as well as the wave components associated with other particles. Thus any particle extraction method must operate on the total reconstruction field U. In numerical reconstruction, the directly transmitted wave B, which is the dc component in the frequency domain, can be easily filtered out.
Furthermore, the correlation term 2 O is negligible as compared with other terms in U. What remains in the total reconstruction field in a numerical reconstruction procedure is the superposition of r i and v i of all particles P i , i =1, 2, . . . , N, where N is the total number of particles recorded. To facilitate our discussion, we decompose the total reconstruction field U as j j U r = +Ω Where r j is the real image wave of particle P j and j Ω represents all other terms in U, which can be written We have shown that r j is real in the in-focus plane of particle P j . However, the imaginary part of U does not vanish due to the addition of j Ω . Therefore, one cannot directly use Im(U) to determine the axial position of particle P j . To overcome this problem, we examine the variance of Im(U), where 2 σ represents the variance of the imaginary part of the complex amplitude. Here the variance is defined in a transverse plane for all pixels that belong to the image of particle P j in that plane. Since the image wave r j is real in the in-focus plane, the variance of its imaginary part, j 2 r σ , should be equal to zero in that plane. On the other hand, in other planes aside from the in-focus plane, the imaginary part of r j is not zero and thus its variance is greater. Therefore, the variance of Im(r) in the neighborhood of a particle has the minimum on the infocus plane. We studied the variance of Im(r) using computer simulation based on Mie scattering and numerical wave propagation [9] As illustrated in Fig. 2a, the forward scattered wave from an opaque particle with diameter (d p =10 µm) is obtained by Mie scattering [6] in a plane consisting of 512 x 512 pixels of size 6.8 µm. In the second phase, the conjugate of the digitized scattered wave is allowed to propagate (numerically) from the recording plane to different transverse planes at various z positions to obtain the real image wave r. In each transverse plane, 2 r σ is calculated for all pixels that belong to the particle image in that plane. Figure 2b shows the variation of 2 r σ with respect to z-z p , the axial distance from the infocus plane. It is seen in the figure that 2 r σ shows a remarkable dipping shape with its minimum (≈zero) at the particle in-focus position.
(a) (b) Fig. 2: (a) Schematic showing the object wave scattered off the particle and the propagate of the real image wave r to the reconstruction planes, (b) variance of Im(r) for a single particle computed in the planes near the particle in-focus plane z=z p . The curve displays the dipping shape with its minimum at the particle depth position To preserve the dipping shape for an arbitrary particle P j in the total reconstruction field change of j 2 σ Ω near the in-focus plane of particle P j must be negligible as compared to the change f )/ z σ ∂ ∂ could be simply obtained from the data shown in Fig. 2b. As a result, we conclude that the dipping characteristic of real image wave r is preserved in the total reconstruction field U. Using that we can identify particles and then extract their depth position.

GENERAL PROCESS FOR EXTRACTING 3D PARTICLE LOCATIONS
Numerous metrology applications involving particle fields (such as velocity field computation from pairs of images and particle tracking in 3D turbulence flow) require a robust and accurate method for extracting particle coordinates from reconstructed 3D images [10,11] . Firstly, let us consider a sample volume of thickness L. This volume corresponds to a set of reconstructed z-planes such that the inequalities z r -L/2 ≤ z ≤ z r +L/2 hold. Figure 3a shows an example of the simulated diffraction pattern of a 30 µm particle located at z 0 =50mm. The reconstructed particle image is shown in Fig. 3b. (a) (b) Fig. 3: Particle image reconstruction. (a) Simulated diffraction pattern of a particle, d=30 µm and z 0 =50 mm and (b) the reconstruction of particle image The general process for determining 3D particle locations within the studied volume is outlined in Fig. 4. Before selecting the supposed particle images by a threshold operation, a background subtraction is performed (step 2). The background is estimated for each z.plane by using a morphological opening operator. The size of the structuring element of this operator must be larger than the particle diameter. The objective of this operation is to improve the image signalto-noise ratio by accentuating the contrast of the reconstructed 2D object images. This operation is followed by a 2D-labelling operation (step 3). Thus, the x-y coordinates of the labelled objects in each plane are available. Fig. 4: General process for determining the 3D particle locations In step 4, we define a 3D-labelled object by clustering the 2D-labelled objects located on the same z-axis. During this operation, we retain only the 3Dlabelled objects that are contained in at least three consecutive planes. The use of this rejection principle is justified by the great depth of field. Indeed, in in-line holography, a given particle image is inevitably detected in several adjacent z-planes. In addition, this operation enables the speckle and other noises to be reduced in the considered volume.
Finally (i.e. in step 5), the result of step 4 is used to recognize the particles among all the 3D-labelled objects in the volume.
For each 3D object, check its record of 2 u σ to see if it has the dipping characteristic. This is to differentiate particles from speckles and other noises in hologram reconstruction. For each 3D object identified as a particle, find the axial position z at which 2 u σ is the minimal through a second-order polynomial regression. The result is saved as the in-focus position of the particle. The radial position of this particle is then copied from the 2D entity that is the closest to the particle's infocus plane.

RESULTS AND DISCUSSION
In order to evaluate the potential of the proposed system, we have used synthesized in-line holograms consisting of 1024×1024 pixels of 9 µm. Particles are randomly distributed with different sizes in a sample volume of depth L. An example of this volume is shown in Fig. 5. An example of a simulated particle field hologram with different particle diameters d=15 µm, 30 µm and 45 µm, is shown in Fig. 6. This pattern is generated with particle number density n s =5 mm¯3 and L =4mm. The numerical reconstruction at different z planes are shown in Fig. 7.
The whole volume (88 z-planes) is reconstructed by using the method described previously. Then, this volume is processed according to the procedure discussed earlier. From Fig. 7 at plane Z r1 the proposed system successively extracted particle field. It is found that all the particles size at that plane with equal size 30 µm. At Z r2 the extracted particles have both sizes of 15µm and 30µm respectively. While at z Z r3 the three different particle sizes are extracted.
From this experiment, it is clear that the proposed system can clearly extract the particles with different sizes even if they are at the same plane. Also, this system can be used for implementing holographic particle tracking with different particle sizes at different z planes.
We have applied the same system in another experiment with the same parameters of first one except that the particles size equal to or slightly larger than the pixels size of the hologram. It is founded that the Z r1 Z r2 Z r3 Fig. 7: The numerical reconstruction of the particle field at different z plane z r1 = 50 mm z r2 = 50.5 mm z r3 = 51 mm intensity of very small particle images cannot be differentiated from the background noise.

CONCLUSION
Digital holography is a promising technique for 3D measurement of small particles. In this study, we have tested the applicability of digital in-line holography to particle field extraction and tracking. Particle information is stored and manipulated in digital forms in the image processing system. Therefore more flexibility is offered to overcome the inherent problems which have been the restrictions to the applications of conventional in-line holography. Theoretical analysis based on Fourier optics and experimental results show that a high lateral resolution and less influence of speckle noise can be achieved in the in-line configuration.
However, its application is often hampered by the poor depth resolution due to insufficient resolving power of the digital image sensors. In this study, we demonstrate that the complex amplitude of the reconstruction field provides a promising solution to this problem. A novel method of particle extraction using complex amplitude has been developed for digital holography of a particle field. This method uses the dipping characteristic of the total reconstruction wave to extract the depth position of opaque particles. Results show that the system outperforms the traditional intensity methods and can predict particle 3D position accurately The significance of these studies is in exploring the effectiveness of the in-line configuration in improving the performance of the digital holography system and thus enhancing its capacity with respect to the higher requirements of resolution and accuracy in micro measurements. Of course, when in-line holography is discussed, another important issue is the effective solutions to the twin-image problem. For successful applications of in-line system, it is a topic deserving more studies. However, digital in-line holography may bring additional information to conventional existing methods for particle field investigation.