Winner-Take-All Neural Network with Massively Optoelectronic Interconnections

: The increased interconnection density, bandwidth, nonlocality and fan-out-fan-in offered by optics over conventional electronic technologies make it a very attractive medium for a variety of application particularity in the field of communication system implementation for all types of computing engines is achieved. This is especially true for neural networks in which the demand for communication resources is extremely high. In this study, the implementation of a neural network that exploits an optical interconnect to perform a real task is described. A pnpn semiconductor device has been connected in parallel with a common load resistance for optical switching. When illuminated, only this device with maximum input will turn on. The voltages across the other devices drop and inhibit their switching ability. With suitable biasing, the winning device can be recall at any time. The result shows, a much faster response (<10ns) can be obtained from thyristors made of III-V compound semiconductors, because their carrier lifetime is considerably shorter than in silicon. With III-V photothyristor, it is possible to combine light emission (even lasing) and photothyristor action in the same unit.


INTRODUCTION
In a winner-take-all function, a collective dynamic competition takes place, which receives the maximum input and suppresses activity in all the other nodes of the network. The mechanism that is responsible for this type of behavior is the competition for a limited resource such as laser resonator gain, current on a bus, or current limited by a common load resistor. The latter is the case considered in this research. This functional unit can be used in a wide variety of applications requiring arbitration. The application of most interest is in competitive neural networks for unsupervised clustering applications, where the winner-take-all is used as a powerful nonlocal non-linearity. In these networks, each node receives a weighted sum of input from a statistical clustered input space. The weight vectors lead to a neuron that represents prototypes of each of the clusters and the largest inner product is related to the prototype to which an input pattern most closely matches. The winner-take-all network selects the largest inner product, corresponding to the best pattern match and assigns class membership.
In this study an optically controlled winner-take-all circuit is described being, based on a pnpn structure that can be used in the optical implementation of a competitive network.
Such a network is implemented as a parallel-optical system that incorporates a diffractive-optical element (DOE). Its performance as a scheduler for both crossbar and self-routing switching fabrics is measured.
In this study, the implementation of a neural network that exploits an optical interconnect to perform a real task. The operation and the design of such a scheduler and operational experimental implementation of an SOF is described. The scheduler uses a neural network in a winner-take-all strategy to optimize decisions on the throughput of Koheen (self organizing).

WTA NEURAL NETWORKS
The basis of the winner-take-all circuit is an electrical network [1] , that has the capability of both lateral and global inhibition. Global inhibition is essential to perform the winner-take-all function and can be realized as a special case of this network, obtained by removing all the local couplings between the cells.
One of the most important uses of this network in Self-Organizing Feature (SOF) mapping in networks is one of the most fascinating topics in the neural network field. Such networks can learn to detect regularities and correlations in their input and adapt their future responses to that input accordingly. The neurons of competitive networks learn to recognize groups of similar input vectors. Self-organizing maps learn to recognize groups of similar input vectors in such a way that neurons physically close together in the neuron layer respond to similar input vectors [2] .
In competitive learning, the neurons in a competitive layer are distributed to recognize frequently presented input vectors. The architecture for a competitive network is shown in Fig. 1, where in this figure the input vector p and the input weight matrix IW 1,1 are accepted to produces a vector having S1 elements. The elements are the negative values of the distances between the input vector and vectors IW 1,1 formed from the rows of the input weight matrix.
The net input n1 of a competitive layer is computed by finding the negative value of distance between input vector p and the weight vectors with the biases b. If all biases are zero, the maximum net input neuron can have 0. This occurs when the input vector p equals the neuron's weight vector. The competitive transfer function accepts a net input vector for a layer and returns neuron output of 0 for all neurons except for the winner, that is the neuron associated with the most positive element of net input n1. The winner's output is 1. If all biases are 0, then the neuron whose weight vector is closest to the input vector has the least negative net input and, therefore, wins the competition in order to output a 1. Biases are used with competitive layers for reasons to be considered later in this study.

OPTOELECTRONIC WTA
To demonstrate the operating principle of the proposed design a commercially available photothyristor is used as active nonlinear device, in the In order to demonstrate the winner-take-all principle in its pure form, it has to be shown that only one node wins the competition regardless of the input. To show this every device is illuminated with sufficient intensity necessary to switch the entire device. After switching-off the light, the competition begins and only that one node with the maximum light input wins the competition and carries nearly all the total current.
In the implementation described here, both a crossbar and a multistage self-routing switching fabric with random-access input queuing is considered. The novelty in this approach is the use of an optoelectronic neural network to perform the input-output matching. The use of neural-network hardware can yield excellent performance on resource-allocation and optimization problems at low cost, is importance is in exploiting analog circuit capabilities and creates a naturally highly parallel approach to the problem. Such a neural network is, however, intractable to be built to any scalable extent in silicon because of the high degree of connectivity required [3] .
In the proposed pnpn switch, the neurons are arranged in a two-dimensional array that represents all possible input-to-output connections such that each neuron corresponds directly to a cross point on the switch Fig. 3 The neuron outputs can vary continuously between the OFF and the ON levels.
The choice of a set of connections requires that the neurons representing all the requested connections are be enabled simultaneously and set to the same intermediate level.
Each neuron has a bias input that tends to increase output but also receives inhibitory input from those neurons that represent blocking connections. The dynamics of the network resolve the conflicts between all the mutually excluded neuron pairs, leaving a valid set of neurons in the ON state and the remainder in the OFF state. The network thus behaves as a winner-take-all (WTA) system with a particularly simple interconnect pattern where each neuron sees only its row and column neighbors, each of which is connected to it by a fixed, inhibitory weight.

SYSTEM IMPLEMENTATION AND SIMULATION RESULT
Here, the basic implementation of the WTA rule in SOF as an optoelectronic array system is described together with system programming and simulation results. The detector array is a commercial photodiode array operated at peak sensitivity with a typical response time of approximately 30 ns. Figure 4 shows schematically DOE (Design of experiment) system that requires two lenses as a design requirement, such that system operates in the Fourier plane. The following describes a typical operational cycle of the neuralswitch scheduler.
To represent the pattern, there must be a two state, "1" when the presence of light, 0 when the light is off. Initially all the lasers are set to a fixed output level that Fig. 4: Schematic of the experimental optical system setup for the crossbar-switch controller is slightly higher than the OFF level. This level sets a stable total power for the array and effectively biases the neurons toward the ON State. When the network is enabled, the lasers of all the requested neurons are connected to their amplifier outputs and the others are set to the OFF level.
Between the laser and the detector arrays are a pair of lenses and a DOE that splits the light from one neuron's laser and focuses it onto the input detectors of the other neurons in the same row and column (for a crossbar) or other required pattern but not to its own input. Because of signal inversion in the amplifier chain, light falling on a detector inhibits the neuron, by decreasing its output. The WTA nature of the setup guarantees a convergence to a feasible solution with those neurons that remain ON.
A vitally critical component of this system in terms of functionality is the DOE. The inhibitory interconnections between the neurons are implemented by the use of far-field scalar DOE's in conjunction with a Fourier lens system. These phase-only diffractive elements are designed by the use of a standard iterative Fourier transform algorithm that is followed by a closed-form iterative technique which is required to produce the uniformity and the signal-to-noise ratio required of the inhibitory interconnections.
By testing this design through MATLAB6 program for multi iteration is found the real pattern from the other stored pattern as follows.
In the first step, the network identifies the winning neuron, then the weights of the winning neuron and the other neurons in its neighborhood, are moved closers to the input vector at each learning step using the selforganizing map learning function (learnsom). The winning neuron weights are altered in proportion to the learning rate. The weights of neurons in its neighborhood are altered in proportion to half the learning rate. The learning rate and the neighborhood Phase 1: (Ordering Phase) This phase lasts for the given number of steps. The neighborhood distance starts as the maximum distance between two neurons and decreases to the tuning neighborhood distance as shown in Fig. 5. The learning rate starts at the orderingphase learning rate and decreases until it reaches the tuning-phase learning rate. As the neighborhood distance and learning rate decrease over this phase, the neurons of the network order themselves typically in the input space with the same topology in which they are ordered physically.
Phase 2: (Tuning Phase) This phase lasts for the rest of training or adaptation. The neighborhood distance stays at the tuning neighborhood distance, which should include only close neighbors (i. e., typically 1. 0). The learning rate continues to decrease from the tuning phase learning rate, but very slowly. The small neighborhood and slowly decreasing learning rate finetune the network, while keeping the ordering learned acquired in the previous phase stable. The number of epochs for the tuning part of training (or time steps for adoption) should be much greater than the number of steps in the ordering phase, because the tuning phase usually takes more iteration.
The weight change dw for a given neuron from the neuron is calculated by learn some input P, activation A2 and learning rate LR: as given below dw = lr*a2*(p'-w) where the activation A2 is found from the layer output A and neuron distances D and the current neighborhood size ND.
Thus, feature maps, while learning to categorize their input, also learn both the topology and distribution of their input. The network for 1000 epochs can be trained using: Fig. 6: An output pattern P through the second training face P = rands(2,100) net. trainParam. epochs = 1000; net = train(net,P); where P is the input pattern This training produces the plot shown in Fig. 6 where the (desired) output pattern is produced from the lattice in the figure for best recognition of the network after 1000 training iteration.

CONCLUSION
In this study, the problems of high interconnection density are solved by using a free-space optical interconnect that exploits diffractive optical techniques to generate the required interconnection patterns and weights. Although in this implementation speed is not a goal, impressive performance in terms of convergence and noise tolerance is observed, implying that scalability is good, so large switch sizes could be utilized for optimizing cost. The pure winner-take-all property of a net made of photothyristors connected in parallel and connected to a common load resistor with a fixed bias which acts as a limited energy resource is demonstrate. The current network described in early version fabricated with rather slow silicon photothyristor with a switching time of several µsec. A much faster response (<10 ns) can be obtained from thyristors made of III-V compound semiconductors, because their carrier lifetime is considerably shorter than in silicon. With III-V photothyristor, it is possible to combine light emission (even lasing) and photothyristor action in the same unit.