nips nips2002 nips2002-23 knowledge-graph by maker-knowledge-mining

23 nips-2002-Adaptive Quantization and Density Estimation in Silicon

Source: pdf

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

Abstract: We present the bump mixture model, a statistical model for analog data where the probabilistic semantics, inference, and learning rules derive from low-level transistor behavior. The bump mixture model relies on translinear circuits to perform probabilistic inference, and floating-gate devices to perform adaptation. This system is low power, asynchronous, and fully parallel, and supports various on-chip learning algorithms. In addition, the mixture model can perform several tasks such as probability estimation, vector quantization, classification, and clustering. We tested a fabricated system on clustering, quantization, and classification of handwritten digits and show performance comparable to the E-M algorithm on mixtures of Gaussians. 1 I n trod u cti on Many system-on-a-chip applications, such as data compression and signal processing, use online adaptation to improve or tune performance. These applications can benefit from the low-power compact design that analog VLSI learning systems can offer. Analog VLSI learning systems can benefit immensely from flexible learning algorithms that take advantage of silicon device physics for compact layout, and that are capable of a variety of learning tasks. One learning paradigm that encompasses a wide variety of learning tasks is density estimation, learning the probability distribution over the input data. A silicon density estimator can provide a basic template for VLSI systems for feature extraction, classification, adaptive vector quantization, and more. In this paper, we describe the bump mixture model, a statistical model that describes the probability distribution function of analog variables using low-level transistor equations. We intend the bump mixture model to be the silicon version of mixture of Gaussians [1], one of the most widely used statistical methods for modeling the probability distribution of a collection of data. Mixtures of Gaussians appear in many contexts from radial basis functions [1] to hidden Markov models [2]. In the bump mixture model, probability computations derive from translinear circuits [3] and learning derives from floating-gate device equations [4]. The bump mixture model can perform different functions such as quantization, probability estimation, and classification. In addition this VLSI mixture model can implement multiple learning algorithms using different peripheral circuitry. Because the equations for system operation and learning derive from natural transistor behavior, we can build large bump mixture model with millions of parameters on a single chip. We have fabricated a bump mixture model, and tested it on clustering, classification, and vector quantization of handwritten digits. The results show that the fabricated system performs comparably to mixtures of Gaussians trained with the E-M algorithm [1]. Our work builds upon several trends of research in the VLSI community. The results in this paper are complement recent work on probability propagation in analog VLSI [5-7]. These previous systems, intended for decoding applications in communication systems, model special forms of probability distributions over discrete variables, and do not incorporate learning. In contrast, the bump mixture model performs inference and learning on probability distributions over continuous variables. The bump mixture model significantly extends previous results on floating-gate circuits [4]. Our system is a fully realized floating-gate learning algorithm that can be used for vector quantization, probability estimation, clustering, and classification. Finally, the mixture model’s architecture is similar to many previous VLSI vector quantizers [8, 9]. We can view the bump mixture model as a VLSI vector quantizer with well-defined probabilistic semantics. Computations such as probability estimation and maximum-likelihood classification have a natural statistical interpretation under the mixture model. In addition, because we rely on floating-gate devices, the mixture model does not require a refresh mechanism unlike previous learning VLSI quantizers. 2 T h e ad ap ti ve b u mp ci rcu i t The adaptive bump circuit [4], depicted in Fig.1(a-b), forms the basis of the bump mixture model. This circuit is slightly different from previous versions reported in the literature. Nevertheless, the high level functionality remains the same; the adaptive bump circuit computes the similarity between a stored variable and an input, and adapts to increase the similarity between the stored variable and input. Fig.1(a) shows the computation portion of the circuit. The bump circuit takes as input, a differential voltage signal (+Vin, −Vin) around a DC bias, and computes the similarity between Vin and a stored value, µ. We represent the stored memory µ as a voltage: µ= Vw- − Vw+ 2 (1) where Vw+ and Vw− are the gate-offset voltages stored on capacitors C1 and C2. Because C1 and C2 isolate the gates of transistors M1 and M2 respectively, these transistors are floating-gate devices. Consequently, the stored voltages Vw+ and Vw− are nonvolatile. We can express the floating-gate voltages Vfg1 and Vfg2 as Vfg1 =Vin +Vw+ and Vfg2 =Vw− −Vin, and the output of the bump circuit as [10]: I out = Ib cosh 2 ( ( 4κ / SU ) (V t fg 1 − V fg 2 ) ) = Ib cosh ( ( 8κ / SU t )(Vin − µ ) ) 2 (2) where Ib is the bias current, κ is the gate-coupling coefficient, Ut is the thermal voltage, and S depends on the transistor sizes. Fig.1(b) shows Iout for three different stored values of µ. As the data show, different µ’s shift the location of the peak response of the circuit. Vw+ V fg1 V in V fg2 Vb M1 −V in M2 I out Vw− C1 C2 V ca sc V2 V1 Vb V tun M6 V fg1 V2 V1 V in j (a) (b) bump circuit's transfer function for three µ's 10 Iout (nA) µ2 µ1 µ3 6 4 2 0 -0.4 -0.2 V fg2 M3 M4 V inj 8 V tun M5 0 V in (c) 0.2 0.4 Figure 1. (a-b) The adaptive bump circuit. (a) The original bump circuit augmented by capacitors C1 and C2, and cascode transistors (driven by Vcasc). (b) The adaptation subcircuit. M3 and M4 control injection on the floating-gates and M5 and M6 control tunneling. (b) Measured output current of a bump circuit for three programmed memories. Fig.1(b) shows the circuit that implements learning in the adaptive bump circuit. We implement learning through Fowler-Nordheim tunneling [11] on tunneling junctions M5-M6 and hot electron injection [12] on the floating-gate transistors M3-M4. Transistor M3 and M5 control injection and tunneling on M1’s floating-gate. Transistors M4 and M6 control injection and tunneling on M2’s floating-gate. We activate tunneling and injection by a high Vtun and low Vinj respectively. In the adaptive bump circuit, both processes increase the similarity between Vin and µ. In addition, the magnitude of the update does not depend on the sign of (Vin − µ) because the differential input provides common-mode rejection to the input differential pair. The similarity function, as seen in Fig.1(b), has a Gaussian-like shape. Consequently, we can equate the output current of the bump circuit with the probability of the input under a distribution parameterized by mean µ: P (Vin | µ ) = I out (3) In addition, increasing the similarity between Vin and µ is equivalent to increasing P(Vin |µ). Consequently, the adaptive bump circuit adapts to maximize the likelihood of the present input under the circuit’s probability distribution. 3 T h e b u mp mi xtu re mod el We now describe the computations and learning rule implemented by the bump mixture model. A mixture model is a general class of statistical models that approximates the probability of an analog input as the weighted sum of probability of the input under several simple distributions. The bump mixture model comprises a set of Gaussian-like probability density functions, each parameterized by a mean vector, µi. Denoting the j th dimension of the mean of the ith density as µij, we express the probability of an input vector x as: P ( x ) = (1/ N ) i P ( x | i ) = (1/ N ) i (∏ P ( x j j | µij ) ) (4) where N is the number of densities in the model and i denotes the ith density. P(x|i) is the product of one-dimensional densities P(xj|µij) that depend on the j th dimension of the ith mean, µij. We derive each one-dimensional probability distribution from the output current of a single bump circuit. The bump mixture model makes two assumptions: (1) the component densities are equally likely, and (2) within each component density, the input dimensions are independent and have equal variance. Despite these restrictions, this mixture model can, in principle, approximate any probability density function [1]. The bump mixture model adapts all µi to maximize the likelihood of the training data. Learning in the bump mixture model is based on the E-M algorithm, the standard algorithm for training Gaussian mixture models. The E-M algorithm comprises two steps. The E-step computes the conditional probability of each density given the input, P(i|x). The M-step updates the parameters of each distribution to increase the likelihood of the data, using P(i|x) to scale the magnitude of each parameter update. In the online setting, the learning rule is: ∆µij = η P (i | x ) ∂ log P ( x j | µij ) ∂µij =η P( x | i) k P( x | k) ∂ log P ( x j | µij ) ∂µij (5) where η is a learning rate and k denotes component densities. Because the adaptive bump circuit already adapts to increase the likelihood of the present input, we approximate E-M by modulating injection and tunneling in the adaptive bump circuit by the conditional probability: ∆µij = η P ( i | x ) f ( x j − µ ij ) (6) where f() is the parameter update implemented by the bump circuit. We can modulate the learning update in (6) with other competitive factors instead of the conditional probability to implement a variety of learning rules such as online K-means. 4 S i l i con i mp l emen tati on We now describe a VLSI system that implements the silicon mixture model. The high level organization of the system detailed in Fig.2, is similar to VLSI vector quantization systems. The heart of the mixture model is a matrix of adaptive bump circuits where the ith row of bump circuits corresponds to the ith component density. In addition, the periphery of the matrix comprises a set of inhibitory circuits for performing probability estimation, inference, quantization, and generating feedback for learning. We send each dimension of an input x down a single column. Unity-gain inverting amplifiers (not pictured) at the boundary of the matrix convert each single ended voltage input into a differential signal. Each bump circuit computes a current that represents (P(xj|µij))σ, where σ is the common variance of the one-dimensional densities. The mixture model computes P(x|i) along the ith row and inhibitory circuits perform inference, estimation, or quantization. We utilize translinear devices [3] to perform all of these computations. Translinear devices, such as the subthreshold MOSFET and bipolar transistor, exhibit an exponential relationship between the gate-voltage and source current. This property allows us to establish a power-law relationship between currents and probabilities (i.e. a linear relationship between gate voltages and log-probabilities). x1 x2 xn Vtun,Vinj P(x|µ11) P(x|µ12) Inh() P(x|µ1n) Output P(x|µ1) µ P(x|µ21) P(x|µ22) P(x|µ2n) Inh() P(x|µ2) µ Figure 2. Bump mixture model architecture. The system comprises a matrix of adaptive bump circuits where each row computes the probability P(x|µi). Inhibitory circuits transform the output of each row into system outputs. Spike generators also transform inhibitory circuit outputs into rate-coded feedback for learning. We compute the multiplication of the probabilities in each row of Fig.2 as addition in the log domain using the circuit in Fig.3 (a). This circuit first converts each bump circuit’s current into a voltage using a diode (e.g. M1). M2’s capacitive divider computes Vavg as the average of the scalar log probabilities, logP(xj|µij): Vavg = (σ / N ) j log P ( x j | µ ij ) (7) where σ is the variance, N is the number of input dimensions, and voltages are in units of κ/Ut (Ut is the thermal voltage and κ is the transistor-gate coupling coefficient). Transistors M2- M5 mirror Vavg to the gate of M5. We define the drain voltage of M5 as log P(x|i) (up to an additive constant) and compute: log ( P ( x | i ) ) = (C1 +C2 ) C1 Vavg = (C1 +C2 )σ C1 N j ( ) log P ( x j | µ ij ) + k (8) where k is a constant dependent on Vg (the control gate voltage on M5), and C1 and C2 are capacitances. From eq.8 we can derive the variance as: σ = NC1 / ( C1 + C2 ) (9) The system computes different output functions and feedback signals for learning by operating on the log probabilities of eq.8. Fig.3(b) demonstrates a circuit that computes P(i|x) for each distribution. The circuit is a k-input differential pair where the bias transistor M0 normalizes currents representing the probabilities P(x|i) at the ith leg. Fig.3(c) demonstrates a circuit that computes P(x). The ith transistor exponentiates logP(x|i), and a single wire sums the currents. We can also apply other inhibitory circuits to the log probabilities such as winner-take-all circuits (WTA) [13] and resistive networks [14]. In our fabricated chip, we implemented probability estimation,conditional probability computation, and WTA. The WTA outputs the index of the most likely component distribution for the present input, and can be used to implement vector quantization and to produce feedback for an online K-means learning rule. At each synapse, the system combines a feedback signal, such as the conditional probability P(i|x), computed at the matrix periphery, with the adaptive bump circuit to implement learning. We trigger adaptation at each bump circuit by a rate-coded spike signal generated from the inhibitory circuit’s current outputs. We generate this spike train with a current-to-spike converter based on Lazzaro’s low-powered spiking neuron [15]. This rate-coded signal toggles Vtun and Vinj at each bump circuit. Consequently, adaptation is proportional to the frequency of the spike train, which is in turn a linear function of the inhibitory feedback signal. The alternative to the rate code would be to transform the inhibitory circuit’s output directly into analog Vs M1 Vavg M2 M5 Vavg C2 ... P(xn|µin)σ P(x1|µi1)σ Vs Vg Vb C1 M4 M3 M0 ... ... log P(x|i) ... ... P(x) P(i|x) log P(x|i) (a) (b) (c) Figure 3. (a) Circuit for computing logP(x|i). (b) Circuit for computing P(i|x). The current through the ith leg represents P(i|x). (c) Circuit for computing P(x). Vtun and Vinj signals. Because injection and tunneling are highly nonlinear functions of Vinj and Vtun respectively, implementing updates that are linear in the inhibitory feedback signal is quite difficult using this approach. 5 E xp eri men tal Res u l ts an d Con cl u s i on s We fabricated an 8 x 8 mixture model (8 probability distribution functions with 8 dimensions each) in a TSMC 0.35µm CMOS process available through MOSIS, and tested the chip on synthetic data and a handwritten digits dataset. In our tests, we found that due to a design error, one of the input dimensions coupled to the other inputs. Consequently, we held that input fixed throughout the tests, effectively reducing the input to 7 dimensions. In addition, we found that the learning rule in eq.6 produced poor performance because the variance of the bump distributions was too large. Consequently, in our learning experiments, we used the hard winner-take-all circuit to control adaptation, resulting in a K-means learning rule. We trained the chip to perform different tasks on handwritten digits from the MNIST dataset [16]. To prepare the data, we first perform PCA to reduce the 784-pixel images to sevendimensional vectors, and then sent the data on-chip. We first tested the circuit on clustering handwritten digits. We trained the chip on 1000 examples of each of the digits 1-8. Fig.4(a) shows reconstructions of the eight means before and after training. We compute each reconstruction by multiplying the means by the seven principal eigenvectors of the dataset. The data shows that the means diverge to associate with different digits. The chip learns to associate most digits with a single probability distribution. The lone exception is digit 5 which doesn’t clearly associate with one distribution. We speculate that the reason is that 3’s, 5’s, and 8’s are very similar in our training data’s seven-dimensional representation. Gaussian mixture models trained with the E-M algorithm also demonstrate similar results, recovering only seven out of the eight digits. We next evaluated the same learned means on vector quantization of a set of test digits (4400 examples of each digit). We compare the chip’s learned means with means learned by the batch E-M algorithm on mixtures of Gaussians (with σ=0.01), a mismatch E-M algorithm that models chip nonidealities, and a non-adaptive baseline quantizer. The purpose of the mismatch E-M algorithm was to assess the effect of nonuniform injection and tunneling strengths in floating-gate transistors. Because tunneling and injection magnitudes can vary by a large amount on different floatinggate transistors, the adaptive bump circuits can learn a mean that is somewhat offcenter. We measured the offset of each bump circuit when adapting to a constant input and constructed the mismatch E-M algorithm by altering the learned means during the M-step by the measured offset. We constructed the baseline quantizer by selecting, at random, an example of each digit for the quantizer codebook. For each quantizer, we computed the reconstruction error on the digit’s seven-dimensional after average squared quantization error before E-M Probability under 7's model (µA) 7 + 9 o 1.5 1 0.5 1 1.5 2 Probability under 9's model (µA) 1 2 3 4 5 6 7 8 digit (b) 2 0.5 10 0 baseline chip E-M/mismatch (a) 2.5 20 2.5 Figure 4. (a) Reconstruction of chip means before and after training with handwritten digits. (b) Comparison of average quantization error on unseen handwritten digits, for the chip’s learned means and mixture models trained by standard algorithms. (c) Plot of probability of unseen examples of 7’s and 9’s under two bump mixture models trained solely on each digit. (c) representation when we represent each test digit by the closest mean. The results in Fig.4(b) show that for most of the digits the chip’s learned means perform as well as the E-M algorithm, and better than the baseline quantizer in all cases. The one digit where the chip’s performance is far from the E-M algorithm is the digit “1”. Upon examination of the E-M algorithm’s results, we found that it associated two means with the digit “1”, where the chip allocated two means for the digit “3”. Over all the digits, the E-M algorithm exhibited a quantization error of 9.98, mismatch E-M gives a quantization error of 10.9, the chip’s error was 11.6, and the baseline quantizer’s error was 15.97. The data show that mismatch is a significant factor in the difference between the bump mixture model’s performance and the E-M algorithm’s performance in quantization tasks. Finally, we use the mixture model to classify handwritten digits. If we train a separate mixture model for each class of data, we can classify an input by comparing the probabilities of the input under each model. In our experiment, we train two separate mixture models: one on examples of the digit 7, and the other on examples of the digit 9. We then apply both mixtures to a set of unseen examples of digits 7 and 9, and record the probability score of each unseen example under each mixture model. We plot the resulting data in Fig.4(c). Each axis represents the probability under a different class. The data show that the model probabilities provide a good metric for classification. Assigning each test example to the class model that outputs the highest probability results in an accuracy of 87% on 2000 unseen digits. Additional software experiments show that mixtures of Gaussians (σ=0.01) trained by the batch E-M algorithm provide an accuracy of 92.39% on this task. Our test results show that the bump mixture model’s performance on several learning tasks is comparable to standard mixtures of Gaussians trained by E-M. These experiments give further evidence that floating-gate circuits can be used to build effective learning systems even though their learning rules derive from silicon physics instead of statistical methods. The bump mixture model also represents a basic building block that we can use to build more complex silicon probability models over analog variables. This work can be extended in several ways. We can build distributions that have parameterized covariances in addition to means. In addition, we can build more complex, adaptive probability distributions in silicon by combining the bump mixture model with silicon probability models over discrete variables [5-7] and spike-based floating-gate learning circuits [4]. A c k n o w l e d g me n t s This work was supported by NSF under grants BES 9720353 and ECS 9733425, and Packard Foundation and Sloan Fellowships. References [1] C. M. Bishop, Neural Networks for Pattern Recognition. Oxford, UK: Clarendon Press, 1995. [2] L. R. Rabiner,

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 edu Abstract We present the bump mixture model, a statistical model for analog data where the probabilistic semantics, inference, and learning rules derive from low-level transistor behavior. [sent-3, score-1.106]

2 The bump mixture model relies on translinear circuits to perform probabilistic inference, and floating-gate devices to perform adaptation. [sent-4, score-1.215]

3 In addition, the mixture model can perform several tasks such as probability estimation, vector quantization, classification, and clustering. [sent-6, score-0.246]

4 We tested a fabricated system on clustering, quantization, and classification of handwritten digits and show performance comparable to the E-M algorithm on mixtures of Gaussians. [sent-7, score-0.303]

5 A silicon density estimator can provide a basic template for VLSI systems for feature extraction, classification, adaptive vector quantization, and more. [sent-12, score-0.201]

6 In this paper, we describe the bump mixture model, a statistical model that describes the probability distribution function of analog variables using low-level transistor equations. [sent-13, score-1.124]

7 We intend the bump mixture model to be the silicon version of mixture of Gaussians [1], one of the most widely used statistical methods for modeling the probability distribution of a collection of data. [sent-14, score-1.216]

8 In the bump mixture model, probability computations derive from translinear circuits [3] and learning derives from floating-gate device equations [4]. [sent-16, score-1.186]

9 The bump mixture model can perform different functions such as quantization, probability estimation, and classification. [sent-17, score-0.971]

10 In addition this VLSI mixture model can implement multiple learning algorithms using different peripheral circuitry. [sent-18, score-0.239]

11 Because the equations for system operation and learning derive from natural transistor behavior, we can build large bump mixture model with millions of parameters on a single chip. [sent-19, score-1.067]

12 We have fabricated a bump mixture model, and tested it on clustering, classification, and vector quantization of handwritten digits. [sent-20, score-1.16]

13 In contrast, the bump mixture model performs inference and learning on probability distributions over continuous variables. [sent-25, score-0.947]

14 The bump mixture model significantly extends previous results on floating-gate circuits [4]. [sent-26, score-1.043]

15 Finally, the mixture model’s architecture is similar to many previous VLSI vector quantizers [8, 9]. [sent-28, score-0.163]

16 We can view the bump mixture model as a VLSI vector quantizer with well-defined probabilistic semantics. [sent-29, score-0.999]

17 Computations such as probability estimation and maximum-likelihood classification have a natural statistical interpretation under the mixture model. [sent-30, score-0.255]

18 In addition, because we rely on floating-gate devices, the mixture model does not require a refresh mechanism unlike previous learning VLSI quantizers. [sent-31, score-0.183]

19 2 T h e ad ap ti ve b u mp ci rcu i t The adaptive bump circuit [4], depicted in Fig. [sent-32, score-1.097]

20 This circuit is slightly different from previous versions reported in the literature. [sent-34, score-0.272]

21 Nevertheless, the high level functionality remains the same; the adaptive bump circuit computes the similarity between a stored variable and an input, and adapts to increase the similarity between the stored variable and input. [sent-35, score-1.338]

22 The bump circuit takes as input, a differential voltage signal (+Vin, −Vin) around a DC bias, and computes the similarity between Vin and a stored value, µ. [sent-38, score-1.263]

23 We represent the stored memory µ as a voltage: µ= Vw- − Vw+ 2 (1) where Vw+ and Vw− are the gate-offset voltages stored on capacitors C1 and C2. [sent-39, score-0.207]

24 Vw+ V fg1 V in V fg2 Vb M1 −V in M2 I out Vw− C1 C2 V ca sc V2 V1 Vb V tun M6 V fg1 V2 V1 V in j (a) (b) bump circuit's transfer function for three µ's 10 Iout (nA) µ2 µ1 µ3 6 4 2 0 -0. [sent-46, score-0.753]

25 (a) The original bump circuit augmented by capacitors C1 and C2, and cascode transistors (driven by Vcasc). [sent-52, score-1.099]

26 (b) Measured output current of a bump circuit for three programmed memories. [sent-55, score-1.018]

27 1(b) shows the circuit that implements learning in the adaptive bump circuit. [sent-57, score-1.061]

28 We implement learning through Fowler-Nordheim tunneling [11] on tunneling junctions M5-M6 and hot electron injection [12] on the floating-gate transistors M3-M4. [sent-58, score-0.565]

29 Transistor M3 and M5 control injection and tunneling on M1’s floating-gate. [sent-59, score-0.294]

30 Transistors M4 and M6 control injection and tunneling on M2’s floating-gate. [sent-60, score-0.294]

31 We activate tunneling and injection by a high Vtun and low Vinj respectively. [sent-61, score-0.272]

32 In the adaptive bump circuit, both processes increase the similarity between Vin and µ. [sent-62, score-0.821]

33 In addition, the magnitude of the update does not depend on the sign of (Vin − µ) because the differential input provides common-mode rejection to the input differential pair. [sent-63, score-0.154]

34 Consequently, we can equate the output current of the bump circuit with the probability of the input under a distribution parameterized by mean µ: P (Vin | µ ) = I out (3) In addition, increasing the similarity between Vin and µ is equivalent to increasing P(Vin |µ). [sent-66, score-1.15]

35 Consequently, the adaptive bump circuit adapts to maximize the likelihood of the present input under the circuit’s probability distribution. [sent-67, score-1.186]

36 3 T h e b u mp mi xtu re mod el We now describe the computations and learning rule implemented by the bump mixture model. [sent-68, score-0.945]

37 A mixture model is a general class of statistical models that approximates the probability of an analog input as the weighted sum of probability of the input under several simple distributions. [sent-69, score-0.431]

38 The bump mixture model comprises a set of Gaussian-like probability density functions, each parameterized by a mean vector, µi. [sent-70, score-1.036]

39 Denoting the j th dimension of the mean of the ith density as µij, we express the probability of an input vector x as: P ( x ) = (1/ N ) i P ( x | i ) = (1/ N ) i (∏ P ( x j j | µij ) ) (4) where N is the number of densities in the model and i denotes the ith density. [sent-71, score-0.295]

40 We derive each one-dimensional probability distribution from the output current of a single bump circuit. [sent-73, score-0.806]

41 The bump mixture model makes two assumptions: (1) the component densities are equally likely, and (2) within each component density, the input dimensions are independent and have equal variance. [sent-74, score-0.969]

42 Despite these restrictions, this mixture model can, in principle, approximate any probability density function [1]. [sent-75, score-0.253]

43 The bump mixture model adapts all µi to maximize the likelihood of the training data. [sent-76, score-0.954]

44 Learning in the bump mixture model is based on the E-M algorithm, the standard algorithm for training Gaussian mixture models. [sent-77, score-1.071]

45 4 S i l i con i mp l emen tati on We now describe a VLSI system that implements the silicon mixture model. [sent-84, score-0.326]

46 The heart of the mixture model is a matrix of adaptive bump circuits where the ith row of bump circuits corresponds to the ith component density. [sent-87, score-2.133]

47 In addition, the periphery of the matrix comprises a set of inhibitory circuits for performing probability estimation, inference, quantization, and generating feedback for learning. [sent-88, score-0.359]

48 Each bump circuit computes a current that represents (P(xj|µij))σ, where σ is the common variance of the one-dimensional densities. [sent-91, score-1.056]

49 The mixture model computes P(x|i) along the ith row and inhibitory circuits perform inference, estimation, or quantization. [sent-92, score-0.569]

50 The system comprises a matrix of adaptive bump circuits where each row computes the probability P(x|µi). [sent-100, score-1.102]

51 Inhibitory circuits transform the output of each row into system outputs. [sent-101, score-0.219]

52 Spike generators also transform inhibitory circuit outputs into rate-coded feedback for learning. [sent-102, score-0.416]

53 2 as addition in the log domain using the circuit in Fig. [sent-104, score-0.337]

54 This circuit first converts each bump circuit’s current into a voltage using a diode (e. [sent-106, score-1.06]

55 We define the drain voltage of M5 as log P(x|i) (up to an additive constant) and compute: log ( P ( x | i ) ) = (C1 +C2 ) C1 Vavg = (C1 +C2 )σ C1 N j ( ) log P ( x j | µ ij ) + k (8) where k is a constant dependent on Vg (the control gate voltage on M5), and C1 and C2 are capacitances. [sent-111, score-0.367]

56 8 we can derive the variance as: σ = NC1 / ( C1 + C2 ) (9) The system computes different output functions and feedback signals for learning by operating on the log probabilities of eq. [sent-113, score-0.232]

57 3(b) demonstrates a circuit that computes P(i|x) for each distribution. [sent-116, score-0.331]

58 The circuit is a k-input differential pair where the bias transistor M0 normalizes currents representing the probabilities P(x|i) at the ith leg. [sent-117, score-0.493]

59 The ith transistor exponentiates logP(x|i), and a single wire sums the currents. [sent-120, score-0.159]

60 We can also apply other inhibitory circuits to the log probabilities such as winner-take-all circuits (WTA) [13] and resistive networks [14]. [sent-121, score-0.404]

61 The WTA outputs the index of the most likely component distribution for the present input, and can be used to implement vector quantization and to produce feedback for an online K-means learning rule. [sent-123, score-0.242]

62 At each synapse, the system combines a feedback signal, such as the conditional probability P(i|x), computed at the matrix periphery, with the adaptive bump circuit to implement learning. [sent-124, score-1.197]

63 We trigger adaptation at each bump circuit by a rate-coded spike signal generated from the inhibitory circuit’s current outputs. [sent-125, score-1.152]

64 This rate-coded signal toggles Vtun and Vinj at each bump circuit. [sent-127, score-0.746]

65 Consequently, adaptation is proportional to the frequency of the spike train, which is in turn a linear function of the inhibitory feedback signal. [sent-128, score-0.184]

66 The alternative to the rate code would be to transform the inhibitory circuit’s output directly into analog Vs M1 Vavg M2 M5 Vavg C2 . [sent-129, score-0.205]

67 Because injection and tunneling are highly nonlinear functions of Vinj and Vtun respectively, implementing updates that are linear in the inhibitory feedback signal is quite difficult using this approach. [sent-150, score-0.417]

68 5 E xp eri men tal Res u l ts an d Con cl u s i on s We fabricated an 8 x 8 mixture model (8 probability distribution functions with 8 dimensions each) in a TSMC 0. [sent-151, score-0.274]

69 35µm CMOS process available through MOSIS, and tested the chip on synthetic data and a handwritten digits dataset. [sent-152, score-0.292]

70 6 produced poor performance because the variance of the bump distributions was too large. [sent-156, score-0.725]

71 Consequently, in our learning experiments, we used the hard winner-take-all circuit to control adaptation, resulting in a K-means learning rule. [sent-157, score-0.294]

72 We trained the chip to perform different tasks on handwritten digits from the MNIST dataset [16]. [sent-158, score-0.346]

73 We first tested the circuit on clustering handwritten digits. [sent-160, score-0.369]

74 We trained the chip on 1000 examples of each of the digits 1-8. [sent-161, score-0.246]

75 The chip learns to associate most digits with a single probability distribution. [sent-166, score-0.276]

76 Gaussian mixture models trained with the E-M algorithm also demonstrate similar results, recovering only seven out of the eight digits. [sent-169, score-0.193]

77 We next evaluated the same learned means on vector quantization of a set of test digits (4400 examples of each digit). [sent-170, score-0.257]

78 01), a mismatch E-M algorithm that models chip nonidealities, and a non-adaptive baseline quantizer. [sent-172, score-0.214]

79 The purpose of the mismatch E-M algorithm was to assess the effect of nonuniform injection and tunneling strengths in floating-gate transistors. [sent-173, score-0.318]

80 Because tunneling and injection magnitudes can vary by a large amount on different floatinggate transistors, the adaptive bump circuits can learn a mean that is somewhat offcenter. [sent-174, score-1.196]

81 We measured the offset of each bump circuit when adapting to a constant input and constructed the mismatch E-M algorithm by altering the learned means during the M-step by the measured offset. [sent-175, score-1.11]

82 We constructed the baseline quantizer by selecting, at random, an example of each digit for the quantizer codebook. [sent-176, score-0.345]

83 For each quantizer, we computed the reconstruction error on the digit’s seven-dimensional after average squared quantization error before E-M Probability under 7's model (µA) 7 + 9 o 1. [sent-177, score-0.185]

84 (a) Reconstruction of chip means before and after training with handwritten digits. [sent-184, score-0.233]

85 (b) Comparison of average quantization error on unseen handwritten digits, for the chip’s learned means and mixture models trained by standard algorithms. [sent-185, score-0.485]

86 (c) Plot of probability of unseen examples of 7’s and 9’s under two bump mixture models trained solely on each digit. [sent-186, score-1.002]

87 4(b) show that for most of the digits the chip’s learned means perform as well as the E-M algorithm, and better than the baseline quantizer in all cases. [sent-189, score-0.266]

88 The one digit where the chip’s performance is far from the E-M algorithm is the digit “1”. [sent-190, score-0.25]

89 Upon examination of the E-M algorithm’s results, we found that it associated two means with the digit “1”, where the chip allocated two means for the digit “3”. [sent-191, score-0.434]

90 The data show that mismatch is a significant factor in the difference between the bump mixture model’s performance and the E-M algorithm’s performance in quantization tasks. [sent-197, score-1.078]

91 Finally, we use the mixture model to classify handwritten digits. [sent-198, score-0.259]

92 If we train a separate mixture model for each class of data, we can classify an input by comparing the probabilities of the input under each model. [sent-199, score-0.288]

93 In our experiment, we train two separate mixture models: one on examples of the digit 7, and the other on examples of the digit 9. [sent-200, score-0.413]

94 We then apply both mixtures to a set of unseen examples of digits 7 and 9, and record the probability score of each unseen example under each mixture model. [sent-201, score-0.415]

95 Our test results show that the bump mixture model’s performance on several learning tasks is comparable to standard mixtures of Gaussians trained by E-M. [sent-210, score-0.955]

96 These experiments give further evidence that floating-gate circuits can be used to build effective learning systems even though their learning rules derive from silicon physics instead of statistical methods. [sent-211, score-0.313]

97 The bump mixture model also represents a basic building block that we can use to build more complex silicon probability models over analog variables. [sent-212, score-1.173]

98 In addition, we can build more complex, adaptive probability distributions in silicon by combining the bump mixture model with silicon probability models over discrete variables [5-7] and spike-based floating-gate learning circuits [4]. [sent-215, score-1.427]

99 Delbruck, "Bump circuits for computing similarity and dissimilarity of analog voltages," California Institute of Technology, CNS Memo 26, 1993. [sent-270, score-0.257]

100 Snow, "Fowler-Nordheim tunneling into thermally grown SiO2," Journal of Applied Physics, vol. [sent-274, score-0.163]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('bump', 0.725), ('circuit', 0.272), ('tunneling', 0.163), ('mixture', 0.163), ('quantization', 0.144), ('circuits', 0.135), ('vlsi', 0.134), ('vin', 0.133), ('chip', 0.13), ('digit', 0.125), ('injection', 0.109), ('silicon', 0.106), ('vavg', 0.098), ('quantizer', 0.091), ('analog', 0.09), ('vw', 0.09), ('transistor', 0.087), ('digits', 0.086), ('ij', 0.083), ('translinear', 0.082), ('handwritten', 0.076), ('transistors', 0.076), ('inhibitory', 0.074), ('voltages', 0.073), ('ith', 0.072), ('vinj', 0.065), ('adaptive', 0.064), ('voltage', 0.063), ('computes', 0.059), ('vtun', 0.057), ('stored', 0.054), ('fabricated', 0.052), ('feedback', 0.05), ('mismatch', 0.046), ('adapts', 0.046), ('unseen', 0.045), ('lazzaro', 0.043), ('logp', 0.043), ('devices', 0.042), ('input', 0.04), ('gaussians', 0.039), ('probability', 0.039), ('baseline', 0.038), ('differential', 0.037), ('comprises', 0.037), ('mixtures', 0.037), ('mp', 0.036), ('consequently', 0.036), ('adaptation', 0.035), ('log', 0.035), ('inh', 0.033), ('similarity', 0.032), ('cmos', 0.031), ('gate', 0.031), ('vb', 0.031), ('density', 0.031), ('classification', 0.031), ('trained', 0.03), ('build', 0.03), ('addition', 0.03), ('ib', 0.029), ('hot', 0.028), ('iout', 0.028), ('seth', 0.028), ('tun', 0.028), ('means', 0.027), ('capacitors', 0.026), ('diorio', 0.026), ('fg', 0.026), ('wta', 0.026), ('implement', 0.026), ('probabilities', 0.025), ('spike', 0.025), ('benefit', 0.024), ('coefficient', 0.024), ('miguel', 0.024), ('periphery', 0.024), ('perform', 0.024), ('thermal', 0.023), ('online', 0.022), ('row', 0.022), ('estimation', 0.022), ('cosh', 0.022), ('mnist', 0.022), ('subthreshold', 0.022), ('control', 0.022), ('associate', 0.021), ('computations', 0.021), ('clustering', 0.021), ('derive', 0.021), ('signal', 0.021), ('parameterized', 0.021), ('densities', 0.021), ('physics', 0.021), ('system', 0.021), ('decoder', 0.021), ('output', 0.021), ('reconstruction', 0.021), ('model', 0.02), ('transform', 0.02)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 1.0 23 nips-2002-Adaptive Quantization and Density Estimation in Silicon

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

2 0.19926825 91 nips-2002-Field-Programmable Learning Arrays

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

Abstract: This paper introduces the Field-Programmable Learning Array, a new paradigm for rapid prototyping of learning primitives and machinelearning algorithms in silicon. The FPLA is a mixed-signal counterpart to the all-digital Field-Programmable Gate Array in that it enables rapid prototyping of algorithms in hardware. Unlike the FPGA, the FPLA is targeted directly for machine learning by providing local, parallel, online analog learning using ﬂoating-gate MOS synapse transistors. We present a prototype FPLA chip comprising an array of reconﬁgurable computational blocks and local interconnect. We demonstrate the viability of this architecture by mapping several learning circuits onto the prototype chip.

3 0.14932525 177 nips-2002-Retinal Processing Emulation in a Programmable 2-Layer Analog Array Processor CMOS Chip

Author: R. Carmona, F. Jiménez-garrido, R. Dominguez-castro, S. Espejo, A. Rodriguez-vázquez

Abstract: A bio-inspired model for an analog programmable array processor (APAP), based on studies on the vertebrate retina, has permitted the realization of complex programmable spatio-temporal dynamics in VLSI. This model mimics the way in which images are processed in the visual pathway, rendering a feasible alternative for the implementation of early vision applications in standard technologies. A prototype chip has been designed and fabricated in a 0.5µm standard CMOS process. Computing power per area and power consumption is amongst the highest reported for a single chip. Design challenges, trade-oﬀs and some experimental results are presented in this paper. 1

4 0.14877781 154 nips-2002-Neuromorphic Bisable VLSI Synapses with Spike-Timing-Dependent Plasticity

Author: Giacomo Indiveri

Abstract: We present analog neuromorphic circuits for implementing bistable synapses with spike-timing-dependent plasticity (STDP) properties. In these types of synapses, the short-term dynamics of the synaptic efﬁcacies are governed by the relative timing of the pre- and post-synaptic spikes, while on long time scales the efﬁcacies tend asymptotically to either a potentiated state or to a depressed one. We fabricated a prototype VLSI chip containing a network of integrate and ﬁre neurons interconnected via bistable STDP synapses. Test results from this chip demonstrate the synapse’s STDP learning properties, and its long-term bistable characteristics.

5 0.14430279 50 nips-2002-Circuit Model of Short-Term Synaptic Dynamics

Author: Shih-Chii Liu, Malte Boegershausen, Pascal Suter

Abstract: We describe a model of short-term synaptic depression that is derived from a silicon circuit implementation. The dynamics of this circuit model are similar to the dynamics of some present theoretical models of shortterm depression except that the recovery dynamics of the variable describing the depression is nonlinear and it also depends on the presynaptic frequency. The equations describing the steady-state and transient responses of this synaptic model ﬁt the experimental results obtained from a fabricated silicon network consisting of leaky integrate-and-ﬁre neurons and different types of synapses. We also show experimental data demonstrating the possible computational roles of depression. One possible role of a depressing synapse is that the input can quickly bring the neuron up to threshold when the membrane potential is close to the resting potential.

6 0.10503758 4 nips-2002-A Differential Semantics for Jointree Algorithms

7 0.085344188 200 nips-2002-Topographic Map Formation by Silicon Growth Cones

8 0.083645098 11 nips-2002-A Model for Real-Time Computation in Generic Neural Microcircuits

9 0.066434585 186 nips-2002-Spike Timing-Dependent Plasticity in the Address Domain

10 0.065037258 52 nips-2002-Cluster Kernels for Semi-Supervised Learning

11 0.061480474 119 nips-2002-Kernel Dependency Estimation

12 0.061380044 51 nips-2002-Classifying Patterns of Visual Motion - a Neuromorphic Approach

13 0.060972512 160 nips-2002-Optoelectronic Implementation of a FitzHugh-Nagumo Neural Model

14 0.054647423 36 nips-2002-Automatic Alignment of Local Representations

15 0.049844876 63 nips-2002-Critical Lines in Symmetry of Mixture Models and its Application to Component Splitting

16 0.047342736 190 nips-2002-Stochastic Neighbor Embedding

17 0.045666475 13 nips-2002-A Note on the Representational Incompatibility of Function Approximation and Factored Dynamics

18 0.044552214 64 nips-2002-Data-Dependent Bounds for Bayesian Mixture Methods

19 0.043641653 73 nips-2002-Dynamic Bayesian Networks with Deterministic Latent Tables

20 0.04325445 128 nips-2002-Learning a Forward Model of a Reflex

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, -0.137), (1, 0.068), (2, -0.012), (3, -0.032), (4, 0.024), (5, 0.237), (6, 0.081), (7, 0.002), (8, 0.018), (9, 0.028), (10, 0.104), (11, 0.004), (12, -0.074), (13, 0.021), (14, 0.149), (15, 0.025), (16, 0.059), (17, -0.067), (18, 0.174), (19, -0.093), (20, 0.311), (21, -0.021), (22, -0.083), (23, 0.009), (24, -0.005), (25, 0.019), (26, 0.039), (27, -0.103), (28, 0.043), (29, 0.12), (30, -0.165), (31, -0.01), (32, -0.179), (33, 0.032), (34, -0.005), (35, 0.06), (36, 0.011), (37, -0.009), (38, -0.035), (39, 0.082), (40, 0.026), (41, -0.096), (42, -0.033), (43, -0.045), (44, 0.001), (45, 0.039), (46, 0.017), (47, -0.101), (48, 0.052), (49, 0.028)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.95033699 23 nips-2002-Adaptive Quantization and Density Estimation in Silicon

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

2 0.92701781 91 nips-2002-Field-Programmable Learning Arrays

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

3 0.71031642 177 nips-2002-Retinal Processing Emulation in a Programmable 2-Layer Analog Array Processor CMOS Chip

Author: R. Carmona, F. Jiménez-garrido, R. Dominguez-castro, S. Espejo, A. Rodriguez-vázquez

4 0.61883956 4 nips-2002-A Differential Semantics for Jointree Algorithms

Author: James D. Park, Adnan Darwiche

Abstract: A new approach to inference in belief networks has been recently proposed, which is based on an algebraic representation of belief networks using multi–linear functions. According to this approach, the key computational question is that of representing multi–linear functions compactly, since inference reduces to a simple process of ev aluating and diﬀerentiating such functions. W e show here that mainstream inference algorithms based on jointrees are a special case of this approach in a v ery precise sense. W e use this result to prov e new properties of jointree algorithms, and then discuss some of its practical and theoretical implications. 1

5 0.5027079 154 nips-2002-Neuromorphic Bisable VLSI Synapses with Spike-Timing-Dependent Plasticity

Author: Giacomo Indiveri

6 0.47815293 50 nips-2002-Circuit Model of Short-Term Synaptic Dynamics

7 0.41315877 160 nips-2002-Optoelectronic Implementation of a FitzHugh-Nagumo Neural Model

8 0.36073911 200 nips-2002-Topographic Map Formation by Silicon Growth Cones

9 0.34222478 11 nips-2002-A Model for Real-Time Computation in Generic Neural Microcircuits

10 0.27280685 63 nips-2002-Critical Lines in Symmetry of Mixture Models and its Application to Component Splitting

11 0.2551918 186 nips-2002-Spike Timing-Dependent Plasticity in the Address Domain

12 0.24549121 131 nips-2002-Learning to Classify Galaxy Shapes Using the EM Algorithm

13 0.23528127 36 nips-2002-Automatic Alignment of Local Representations

14 0.23051509 13 nips-2002-A Note on the Representational Incompatibility of Function Approximation and Factored Dynamics

15 0.22660734 44 nips-2002-Binary Tuning is Optimal for Neural Rate Coding with High Temporal Resolution

16 0.21782617 52 nips-2002-Cluster Kernels for Semi-Supervised Learning

17 0.21487065 138 nips-2002-Manifold Parzen Windows

18 0.21329746 190 nips-2002-Stochastic Neighbor Embedding

19 0.2075216 111 nips-2002-Independent Components Analysis through Product Density Estimation

20 0.20224734 22 nips-2002-Adaptive Nonlinear System Identification with Echo State Networks

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(3, 0.018), (6, 0.038), (23, 0.021), (42, 0.057), (51, 0.25), (54, 0.093), (55, 0.036), (57, 0.013), (67, 0.021), (68, 0.033), (74, 0.062), (83, 0.132), (92, 0.036), (98, 0.074)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.79470313 23 nips-2002-Adaptive Quantization and Density Estimation in Silicon

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

2 0.77051806 101 nips-2002-Handling Missing Data with Variational Bayesian Learning of ICA

Author: Kwokleung Chan, Te-Won Lee, Terrence J. Sejnowski

Abstract: Missing data is common in real-world datasets and is a problem for many estimation techniques. We have developed a variational Bayesian method to perform Independent Component Analysis (ICA) on high-dimensional data containing missing entries. Missing data are handled naturally in the Bayesian framework by integrating the generative density model. Modeling the distributions of the independent sources with mixture of Gaussians allows sources to be estimated with different kurtosis and skewness. The variational Bayesian method automatically determines the dimensionality of the data and yields an accurate density model for the observed data without overﬁtting problems. This allows direct probability estimation of missing values in the high dimensional space and avoids dimension reduction preprocessing which is not feasible with missing data.

3 0.67024863 191 nips-2002-String Kernels, Fisher Kernels and Finite State Automata

Author: Craig Saunders, Alexei Vinokourov, John S. Shawe-taylor

Abstract: In this paper we show how the generation of documents can be thought of as a k-stage Markov process, which leads to a Fisher kernel from which the n-gram and string kernels can be re-constructed. The Fisher kernel view gives a more flexible insight into the string kernel and suggests how it can be parametrised in a way that reflects the statistics of the training corpus. Furthermore, the probabilistic modelling approach suggests extending the Markov process to consider sub-sequences of varying length, rather than the standard fixed-length approach used in the string kernel. We give a procedure for determining which sub-sequences are informative features and hence generate a Finite State Machine model, which can again be used to obtain a Fisher kernel. By adjusting the parametrisation we can also influence the weighting received by the features . In this way we are able to obtain a logarithmic weighting in a Fisher kernel. Finally, experiments are reported comparing the different kernels using the standard Bag of Words kernel as a baseline. 1

4 0.62725306 91 nips-2002-Field-Programmable Learning Arrays

Author: Seth Bridges, Miguel Figueroa, Chris Diorio, Daniel J. Hsu

5 0.62053484 200 nips-2002-Topographic Map Formation by Silicon Growth Cones

Author: Brian Taba, Kwabena A. Boahen

Abstract: We describe a self-configuring neuromorphic chip that uses a model of activity-dependent axon remodeling to automatically wire topographic maps based solely on input correlations. Axons are guided by growth cones, which are modeled in analog VLSI for the first time. Growth cones migrate up neurotropin gradients, which are represented by charge diffusing in transistor channels. Virtual axons move by rerouting address-events. We refined an initially gross topographic projection by simulating retinal wave input. 1 Neuromorphic Systems Neuromorphic engineers are attempting to match the computational efficiency of biological systems by morphing neurocircuitry into silicon circuits [1]. One of the most detailed implementations to date is the silicon retina described in [2] . This chip comprises thirteen different cell types, each of which must be individually and painstakingly wired. While this circuit-level approach has been very successful in sensory systems, it is less helpful when modeling largely unelucidated and exceedingly plastic higher processing centers in cortex. Instead of an explicit blueprint for every cortical area, what is needed is a developmental rule that can wire complex circuits from minimal specifications. One candidate is the famous

6 0.60609174 168 nips-2002-Real-Time Monitoring of Complex Industrial Processes with Particle Filters

7 0.58770108 110 nips-2002-Incremental Gaussian Processes

8 0.58288312 130 nips-2002-Learning in Zero-Sum Team Markov Games Using Factored Value Functions

9 0.52207261 177 nips-2002-Retinal Processing Emulation in a Programmable 2-Layer Analog Array Processor CMOS Chip

10 0.48868388 154 nips-2002-Neuromorphic Bisable VLSI Synapses with Spike-Timing-Dependent Plasticity

11 0.48339981 119 nips-2002-Kernel Dependency Estimation

12 0.48233867 50 nips-2002-Circuit Model of Short-Term Synaptic Dynamics

13 0.48192155 55 nips-2002-Combining Features for BCI

14 0.48150864 37 nips-2002-Automatic Derivation of Statistical Algorithms: The EM Family and Beyond

15 0.47673893 186 nips-2002-Spike Timing-Dependent Plasticity in the Address Domain

16 0.47667646 11 nips-2002-A Model for Real-Time Computation in Generic Neural Microcircuits

17 0.47661847 85 nips-2002-Fast Kernels for String and Tree Matching

18 0.47022814 10 nips-2002-A Model for Learning Variance Components of Natural Images

19 0.46993935 68 nips-2002-Discriminative Densities from Maximum Contrast Estimation

20 0.46939257 21 nips-2002-Adaptive Classification by Variational Kalman Filtering