nips nips2011 nips2011-11 knowledge-graph by maker-knowledge-mining

11 nips-2011-A Reinforcement Learning Theory for Homeostatic Regulation

Source: pdf

Author: Mehdi Keramati, Boris S. Gutkin

Abstract: Reinforcement learning models address animal’s behavioral adaptation to its changing “external” environment, and are based on the assumption that Pavlovian, habitual and goal-directed responses seek to maximize reward acquisition. Negative-feedback models of homeostatic regulation, on the other hand, are concerned with behavioral adaptation in response to the “internal” state of the animal, and assume that animals’ behavioral objective is to minimize deviations of some key physiological variables from their hypothetical setpoints. Building upon the drive-reduction theory of reward, we propose a new analytical framework that integrates learning and regulatory systems, such that the two seemingly unrelated objectives of reward maximization and physiological-stability prove to be identical. The proposed theory shows behavioral adaptation to both internal and external states in a disciplined way. We further show that the proposed framework allows for a uniﬁed explanation of some behavioral pattern like motivational sensitivity of different associative learning mechanism, anticipatory responses, interaction among competing motivational systems, and risk aversion.

Reference: text

Summary: the most important sentenses genereted by tfidf model

sentIndex sentText sentNum sentScore

1 fr Abstract Reinforcement learning models address animal’s behavioral adaptation to its changing “external” environment, and are based on the assumption that Pavlovian, habitual and goal-directed responses seek to maximize reward acquisition. [sent-5, score-0.62]

2 The proposed theory shows behavioral adaptation to both internal and external states in a disciplined way. [sent-8, score-0.42]

3 We further show that the proposed framework allows for a uniﬁed explanation of some behavioral pattern like motivational sensitivity of different associative learning mechanism, anticipatory responses, interaction among competing motivational systems, and risk aversion. [sent-9, score-1.347]

4 1 Introduction “Reinforcement learning” and “negative-feedback models of homeostatic regulation” are two control theory-based computational frameworks that have had major contributions in learning and motivation literatures in behavioral psychology, respectively. [sent-10, score-0.557]

5 Proposing neurobiologically-plausible algorithms, computational theory of reinforcement learning (RL) explains how animals adapt their behavior to varying contingencies in their “external” environment, through persistently updating their estimates of the rewarding value of feasible choices [1]. [sent-11, score-0.385]

6 Critically, this theory is built upon one hypothetical assumption: animals behavioral objective is to maximize reward acquisition. [sent-13, score-0.556]

7 In this respect, without addressing the question of how the brain constructs reward, the RL theory gives a normative explanation for how the brains decision making circuitry shapes instrumental responses as a function of external cues, so as the animal to satisfy its reward-maximization objective. [sent-14, score-0.515]

8 Negative-feedback models of homeostatic regulation (HR), on the other hand, seek to explain regular variabilities in behavior as a function of the internal state of the animal, when the external stimuli are ﬁxed [3, 4]. [sent-15, score-0.738]

9 Correspondingly, behavioral homeostasis refers to corrective responses that are triggered by deviation of 1 a regulated variable from its hypothetical setpoint. [sent-17, score-0.444]

10 (2) Intravenous (and even intragastric, in some cases) injection of food is not motivating, even though it alleviates deviation of the motivational state from its homeostatic setpoint. [sent-22, score-1.151]

11 These behavioral patterns are used as evidence to argue that maintaining homeostasis (reducing the negative feedback) is not a behavioral objective. [sent-24, score-0.494]

12 (3) The traditional homeostatic regulation theory simply assumes that the animal knows how to translate its various physiological deﬁcits into the appropriate behaviors. [sent-26, score-0.881]

13 In fact, without taking into account the contextual state of the animal, negative feedback models only address the question of whether or not the animal should have motivation toward a certain outcome, without answering how the outcome can be achieved through a series of instrumental actions. [sent-27, score-0.623]

14 The existence of these shortcomings calls for rethinking the traditional view toward homeostatic regulation theory. [sent-28, score-0.42]

15 We believe that these shortcomings, as well as the weak spot of RL models in not taking into account the internal drive state of the animal, all arise from lack of an integrated view toward learning and motivation. [sent-29, score-0.419]

16 2 The model The term “reward” (with many equivalents like reinforcer, motivational salience, utility, etc. [sent-32, score-0.452]

17 In purely behavioral terms, it refers to a stimulus delivered to the animal after a response is made, that increases the probability of making that response in the future. [sent-34, score-0.56]

18 rt denotes the rewarding value of the outcome the animal receives at time t, which is a often set to a “ﬁxed” value that can be either positive or negative, depending on whether the corresponding stimulus is appetitive or aversive, respectively. [sent-41, score-0.627]

19 However, animals motivation for outcomes is not ﬁxed, but a function of their internal state: a food pellet is more rewarding to a hungry, than a sated rat. [sent-42, score-0.857]

20 In fact, the internal state (also referred to as drive, or motivational state) of the animal affects the reinforcing value of a constant external outcome. [sent-43, score-1.111]

21 As an attempt to identify the nature of reward and its relation to drive states, neo-behaviorists like Hull [8], Spence, and Mowrer have proposed the “drive reduction” theory of motivation. [sent-44, score-0.428]

22 According to this theory, one primary mechanism underlying reward is drive reduction. [sent-45, score-0.428]

23 In terms of homeostatic 2 regulation theory, reward is deﬁned as a stimulus that reduces the discrepancy between the current and the desired drive state of the animal, i. [sent-46, score-0.924]

24 e, food pellet is rewarding to a hungry rat because it fulﬁlls a physiological need. [sent-47, score-0.739]

25 , hN,t } denote the physiological state of the animal at time t, and H ∗ = {h∗ , h∗ , . [sent-50, score-0.537]

26 As a special case, ﬁg1 2 N ure 1 shows a simpliﬁed system where food and water constitute all the physiological needs. [sent-53, score-0.511]

27 The homeostatic space, as a multidimensional metric space, is a hypothetical construct that allow us to explain a wide range of behavioral and physiological evidence in a uniﬁed framework. [sent-56, score-0.726]

28 Since animals physiological states are most often below the setpoint, our focus in this paper is mostly on the quarter of the homeostatic space that is below the homeostatic setpoint. [sent-57, score-0.859]

29 Assume that as the result of some actions, the animal receives an outcome at time t that contains ki,t units of the constituent hi , for each i ∈ {1, 2, . [sent-59, score-0.4]

30 Consumption of that outcome will result in a transition of the physiological state from Ht to Ht+1 = Ht + Kt , and consequently, a transition of the drive state from d(Ht ) to d(Ht + Kt ). [sent-65, score-0.645]

31 , 0) ,when the animal is in the motivational state Ht , will have the four below properties. [sent-73, score-0.822]

32 1 Reward value increases with the dose of outcome The reward function is an increasing function of the magnitude of the outcome (e. [sent-76, score-0.443]

33 2 Excitatory effect of deprivation level Increasing the deprivation level of the animal will increase the reinforcing strength of a constant dose of a corresponding outcome, i. [sent-83, score-0.593]

34 a single pellet of food is more rewarding to a hungry, than a sated rate. [sent-85, score-0.559]

35 dr(Ht , Kt ) > 0 : for kj,t > 0 (5) d|h∗ − hj,t | j Consistently, the level of food deprivation in rats is demonstrated to increases the breakpoint in progressive ratio schedules [10]. [sent-86, score-0.458]

36 3 Inhibitory effect of the irrelevant drive A large body of behavioral experiments shows that deprivation level for one need has an inhibitory effect on the reinforcing value of outcomes that satisfy irrelevant needs. [sent-88, score-0.753]

37 It is well known that high levels of irrelevant thirst impairs Pavlovian as well as instrumental responses for food during both acquisition and extinction. [sent-90, score-0.415]

38 Reciprocally, food deprivation is demonstrated to suppress Pavlovian and instrumental water-related responses. [sent-91, score-0.456]

39 This behavioral pattern can be interpreted as competition among different motivational systems (different dimensions of the homeostatic space), and is consistent with the notion of “complementary” relation between some goods in economics, like between cake and coffee. [sent-94, score-0.988]

40 It is clear that the four mentioned properties of the reward function depend on the speciﬁc functional form adopted for the drive function (equation 2). [sent-102, score-0.428]

41 In fact, the drive-reduction deﬁnition of reward allows for the validity of the form of drive function to be experimentally tested in behavioral tasks. [sent-103, score-0.64]

42 4 Homeostatic control through learning mechanisms Despite signiﬁcant day-to-day variations in energy expenditure and food availability, the homeostatic regulation system involved in control of food-intake and energy balance is a remarkably precise system. [sent-104, score-0.774]

43 This adaptive nature of homeostatic regulation system can be achieved by employing the animals learning mechanisms, which are capable of adopting ﬂexible behavioral strategies to cope with changes in the environmental conditions. [sent-105, score-0.748]

44 In this section we are interested in looking at behavioral and theoretical implications of interaction between homeostatic regulation and reward learning systems. [sent-106, score-0.84]

45 One theoretical implication of the proposed deﬁnition of reward is that it reconciles the RL and homeostatic regulation theories, in terms of normative assumptions behind them. [sent-107, score-0.666]

46 Since the reward function deﬁned by equation 3 depends on the internal state and thus, is nonstationary, some appropriate adjustments in the classical RL algorithms that try to ﬁnd an optimal policy must be thought of. [sent-110, score-0.461]

47 Consistent with this view, some behavioral studies show that internal state can work in the same way that external state works. [sent-112, score-0.547]

48 That is, animals are able to acquire responses conditioned upon some certain motivational states (e. [sent-113, score-0.6]

49 Moreover, since the next external state only depends on the current external but not internal state, such an augmented MDP will have signiﬁcant redundant structure. [sent-117, score-0.319]

50 Beside this algorithm-independent expansion of state-space argued above, the proposed deﬁnition of reward provides an opportunity for discussing how different associate learning systems in the brain take the animal’s internal state into account. [sent-119, score-0.451]

51 Here we discuss motivational sensitivity in habitual (hypothesized to use a model-free RL, like temporal difference algorithm [13]), goal-directed (hypothesized to use a model-based RL [13]), and Pavlovian learning systems. [sent-120, score-0.593]

52 In our framework, as long as a model-based algorithm is involved in decision making, all that the animal needs to do when its internal state shifts is to update the incentive value (reward function) of each potential outcome. [sent-122, score-0.522]

53 The reward function being updated, the animal will be able to take the optimal policy, given that the state-transition function is learned completely. [sent-123, score-0.502]

54 In order to update the reward function, one way is that the animal re-experiences outcomes in its new motivational state. [sent-124, score-0.995]

55 This way seems to be how the goal-directed system works, since re-exposure is demonstrated to be necessary in order for the animals’ behavior to be sensitive to changes in motivational state [11]. [sent-125, score-0.617]

56 The second way is to update the reward function without re-exposure, but through directly estimating the drivereduction effect of outcomes in the current motivational state, using equation 3. [sent-126, score-0.761]

57 This Pavlovian-type behavioral adaptation is observed even when the animal has never experienced the outcome in its current motivational state during its whole life. [sent-128, score-1.165]

58 Furthermore, the fact that this motivational sensitivity is outcome-speciﬁc shows that the underlying associative learning structure is model-based (i. [sent-130, score-0.51]

59 without new learning) adapt the animal’s behavioral response after shifts in the motivational state [11]. [sent-135, score-0.767]

60 It might be concluded from this observation that action-values in the habitual system only depend on the internal state of the animal during the course of learning (past experiences), but not performance. [sent-136, score-0.636]

61 Thus, the habitual system doesn’t know what speciﬁc action-values should be updated when a novel internal state is being experienced. [sent-138, score-0.342]

62 However, it has been argued by some authors [14] that even though habitual responses are insensitive to sensory-speciﬁc satiety, they are sensitive to motivational manipulations in a general, outcomeindependent way. [sent-139, score-0.671]

63 Note that the previous form of motivational insensitivity concerns lack of behavioral adaptation in an outcome-speciﬁc way (for example, lack of greater preference for food-seeking behavior, compared to water-seeking behavior, after a shift from satiety to hunger state). [sent-140, score-0.806]

64 Borrowing from the Hullian concept of “generalized drive” [8], it has been proposed that transition from an outcome-sated to an outcome-deprived motivational state will energize all pre-potent habitual responses, irrespective of whether or not those actions result in that certain outcome [14]. [sent-141, score-0.751]

65 For example, a motivational shift from satiety to hunger will result in energization of both food-seeking and water-seeking habitual responses. [sent-142, score-0.686]

66 Taking a normative perspective, we argue here that this outcome-independent energizing effect is an approximate way of updating the value of state-action pairs when the motivational state shifts instantaneously. [sent-143, score-0.642]

67 Assuming that the animal is trained under the ﬁxed internal state H0 , and then tested in a novel internal state H1 , the cached values in the habitual system can be approximated in the new motivational state by: Q1 (s, a) = d(H1 ) . [sent-144, score-1.363]

68 According to this update rule, all the prepotent actions will get energized if deviation from the homeostatic setpoint increases in the new internal state, whether or not the outcome of those actions are more desired in the new state. [sent-146, score-0.623]

69 This means that the energizing effect will result in rational behavior after motivational shift, only when the new internal state is an 6 ampliﬁcation or abridge of the old internal state in all the dimension of the homeostatic space, with equal magnitude. [sent-149, score-1.258]

70 Since the model-free system does not learn the causal model of the environment, it cannot show motivational sensitivity in an outcome-speciﬁc way, and this general energizing effect is the best approximate way to react to motivational manipulations. [sent-150, score-1.03]

71 Anticipatory eating and drinking, as two examples, are deﬁned by taking food and water in the absence of any detectable decrease in the corresponding physiological signals. [sent-153, score-0.517]

72 Although it is quite clear that explaining such behavioral patterns requires integrating homeostatic mechanisms and learning (as predictive) processes, to our knowledge, no well-deﬁned mechanism has been proposed for it so far. [sent-154, score-0.549]

73 We show here, through an example, how the proposed model can reconcile anticipatory responses to the homeostatic regulation theory. [sent-155, score-0.651]

74 Body temperature, which has long been a topic of interest in the homeostatic regulation literature, is shown to increase back to its normal level by shivering, after the animal is placed into a cold place. [sent-157, score-0.746]

75 Interestingly, cues that predict being placed into a cold place induce anticipatory shivering and result in the body temperature to go above the normal temperature of the animal [15]. [sent-158, score-0.733]

76 This behavior is interpreted as an attempt by the animal to alleviate the severity of deviation from the setpoint. [sent-160, score-0.325]

77 To model this example, lets assume that x∗ is the normal temperature of the animal, and that putting the animal in the cold place will result in a decrease of lx units in the body temperature. [sent-161, score-0.484]

78 Furthermore, assume that when presented with the coldness-predicting cue (SC ), the animal chooses how much to shiver and thus, increases its body temperature by kx units. [sent-162, score-0.489]

79 In a scenario like this, after observing the coldness-predicting cue, the animals temperature will shift from x∗ to x∗ + kx , as a result of anticipatory shivering. [sent-163, score-0.447]

80 Assuming that the animal will then experience coldness after a delay of one time-unit, its temperature will transit from x∗ + kx to x∗ + kx − lx . [sent-164, score-0.591]

81 Finally, by assuming that after one more time-unit the body temperature will go back to the normal level, x∗ , the animal will receive another reward, discounted with the rate of γ 2 . [sent-166, score-0.45]

82 If the discount factor is one, then regardless of the trajectory of the motivational state, the sum of rewards for all policies that start from a certain homeostatic point and ﬁnish at another point, will be equal. [sent-173, score-0.824]

83 In that case, the sum of rewards for all values of kx in the anticipatory shivering example will be zero and thus, anticipatory strategies will not be preferred to a reactive-homeostasis strategy. [sent-174, score-0.533]

84 through pharmacological agents known to modulate discount rate) should increase the probability that the animal show an anticipatory response, and vice versa. [sent-177, score-0.491]

85 7 5 Discussion Despite considerable differences, some common principles are argued to govern all motivational systems. [sent-178, score-0.496]

86 we have proposed a model that captures some of these commonalities between homoestatically regulated motivational systems. [sent-179, score-0.51]

87 We showed how the rewarding value of an outcome can be computed as a function of the animal’s internal state and the constituting elements of the outcome, through the drive reduction equation. [sent-180, score-0.749]

88 We further demonstrated how this computed reward can be used by different learning mechanisms to form appropriate Pavlovian or instrumental associations. [sent-181, score-0.341]

89 Such a long delay between an instrumental response and its consequent drive reduction effect (reward) would make it difﬁcult for an appropriate association to be established. [sent-183, score-0.395]

90 (2) Dopamine neurons, which are supposed to carry reward learning signal, are demonstrated to show instantaneous burst activity in response to unexpected food rewards, without waiting for the food to be digested, and drive to be reduced [16]. [sent-185, score-1.071]

91 (3) Intravenous injection (and intragastric intubation, in some cases) of food is not rewarding, even though its drive reduction effect is equal to when that food is ingested orally. [sent-186, score-0.934]

92 Making the story even more complicated, this taste-dependent rewarding value of food is demonstrated to be modulated by not only the approximated nutritional content of the food, but also the internal physiological state of the animal [16]. [sent-191, score-1.224]

93 However, assuming that taste and other sensory properties of a food outcome give an estimate, kˆ , 1,t of its true nutritional content, k1,t , the rewarding effect of food can be approximated by equation 3, as soon as the food is sensed, or taken. [sent-192, score-1.309]

94 It explains that gastric, olfactory, or visual information of food is necessary for its reinforcing effect to be induced and thus, intravenous injection of food is not reinforcing due to lack of appropriate sensory information. [sent-195, score-0.929]

95 Moreover, there is no delay in this mechanism between food intake and its drive-reduction effect and therefore, Dopamine neurons can respond instantaneously. [sent-196, score-0.328]

96 Finally, as equation 3 predicts, this taste-dependent rewarding value is modulated by the motivational state of the animal. [sent-197, score-0.752]

97 Previous computational accounts in the psychological literature attempting to incorporate internalstate dependence of motivation into the RL models use ad hoc addition or multiplication of drive state with the a priori reward magnitude [17, 13]. [sent-198, score-0.548]

98 A more recent work [12] also uses RL framework where reward is generated by drive difference. [sent-201, score-0.428]

99 Apart from physiological and behavioral plausibility, the theoretical novelty of our proposed framework is in formalizing the hypothetical concept of drive, as a mapping from physiological to motivational state. [sent-203, score-1.044]

100 Reward effects of food via stomach ﬁstula compared with those of food via mouth. [sent-252, score-0.584]

similar papers computed by tfidf model

tfidf for this paper:

wordName wordTfidf (topN-words)

[('motivational', 0.452), ('homeostatic', 0.301), ('animal', 0.294), ('food', 0.292), ('drive', 0.22), ('behavioral', 0.212), ('reward', 0.208), ('rewarding', 0.2), ('anticipatory', 0.173), ('physiological', 0.167), ('internal', 0.123), ('regulation', 0.119), ('habitual', 0.117), ('rl', 0.116), ('outcome', 0.106), ('reinforcing', 0.106), ('ht', 0.106), ('pavlovian', 0.093), ('setpoint', 0.093), ('animals', 0.09), ('kx', 0.087), ('kt', 0.077), ('state', 0.076), ('temperature', 0.073), ('homeostasis', 0.07), ('deprivation', 0.067), ('instrumental', 0.065), ('external', 0.06), ('regulated', 0.058), ('responses', 0.058), ('hunger', 0.053), ('shivering', 0.053), ('regulatory', 0.053), ('lx', 0.05), ('discounted', 0.048), ('drinking', 0.047), ('rewards', 0.047), ('hypothetical', 0.046), ('argued', 0.044), ('motivation', 0.044), ('outcomes', 0.041), ('energizing', 0.04), ('hungry', 0.04), ('intragastric', 0.04), ('intravenous', 0.04), ('nutritional', 0.04), ('pellet', 0.04), ('satiety', 0.04), ('feedback', 0.038), ('normative', 0.038), ('progressive', 0.038), ('drives', 0.037), ('mechanisms', 0.036), ('psychology', 0.036), ('effect', 0.036), ('contingencies', 0.035), ('dopamine', 0.035), ('body', 0.035), ('associative', 0.034), ('demonstrated', 0.032), ('cold', 0.032), ('eating', 0.032), ('behavior', 0.031), ('injection', 0.03), ('policy', 0.03), ('reinforcement', 0.029), ('incentive', 0.029), ('rats', 0.029), ('stimuli', 0.028), ('sensory', 0.027), ('response', 0.027), ('dr', 0.027), ('animats', 0.027), ('appetitive', 0.027), ('aversion', 0.027), ('depletion', 0.027), ('gallistel', 0.027), ('lnc', 0.027), ('milk', 0.027), ('physiol', 0.027), ('reinforcer', 0.027), ('sated', 0.027), ('water', 0.026), ('system', 0.026), ('adaptation', 0.025), ('discount', 0.024), ('sensitivity', 0.024), ('shift', 0.024), ('equation', 0.024), ('reduction', 0.024), ('appetite', 0.023), ('consequent', 0.023), ('dose', 0.023), ('emotion', 0.023), ('goods', 0.023), ('niv', 0.023), ('unconditioned', 0.023), ('uni', 0.023), ('agent', 0.023), ('hypothesized', 0.023)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.99999982 11 nips-2011-A Reinforcement Learning Theory for Homeostatic Regulation

Author: Mehdi Keramati, Boris S. Gutkin

2 0.19350421 88 nips-2011-Environmental statistics and the trade-off between model-based and TD learning in humans

Author: Dylan A. Simon, Nathaniel D. Daw

Abstract: There is much evidence that humans and other animals utilize a combination of model-based and model-free RL methods. Although it has been proposed that these systems may dominate according to their relative statistical efﬁciency in different circumstances, there is little speciﬁc evidence — especially in humans — as to the details of this trade-off. Accordingly, we examine the relative performance of different RL approaches under situations in which the statistics of reward are differentially noisy and volatile. Using theory and simulation, we show that model-free TD learning is relatively most disadvantaged in cases of high volatility and low noise. We present data from a decision-making experiment manipulating these parameters, showing that humans shift learning strategies in accord with these predictions. The statistical circumstances favoring model-based RL are also those that promote a high learning rate, which helps explain why, in psychology, the distinction between these strategies is traditionally conceived in terms of rulebased vs. incremental learning. 1

3 0.13689879 163 nips-2011-MAP Inference for Bayesian Inverse Reinforcement Learning

Author: Jaedeug Choi, Kee-eung Kim

Abstract: The difﬁculty in inverse reinforcement learning (IRL) arises in choosing the best reward function since there are typically an inﬁnite number of reward functions that yield the given behaviour data as optimal. Using a Bayesian framework, we address this challenge by using the maximum a posteriori (MAP) estimation for the reward function, and show that most of the previous IRL algorithms can be modeled into our framework. We also present a gradient method for the MAP estimation based on the (sub)differentiability of the posterior distribution. We show the effectiveness of our approach by comparing the performance of the proposed method to those of the previous algorithms. 1

4 0.12668785 130 nips-2011-Inductive reasoning about chimeric creatures

Author: Charles Kemp

Abstract: Given one feature of a novel animal, humans readily make inferences about other features of the animal. For example, winged creatures often ﬂy, and creatures that eat ﬁsh often live in the water. We explore the knowledge that supports these inferences and compare two approaches. The ﬁrst approach proposes that humans rely on abstract representations of dependency relationships between features, and is formalized here as a graphical model. The second approach proposes that humans rely on speciﬁc knowledge of previously encountered animals, and is formalized here as a family of exemplar models. We evaluate these models using a task where participants reason about chimeras, or animals with pairs of features that have not previously been observed to co-occur. The results support the hypothesis that humans rely on explicit representations of relationships between features. Suppose that an eighteenth-century naturalist learns about a new kind of animal that has fur and a duck’s bill. Even though the naturalist has never encountered an animal with this pair of features, he should be able to make predictions about other features of the animal—for example, the animal could well live in water but probably does not have feathers. Although the platypus exists in reality, from a eighteenth-century perspective it qualiﬁes as a chimera, or an animal that combines two or more features that have not previously been observed to co-occur. Here we describe a probabilistic account of inductive reasoning and use it to account for human inferences about chimeras. The inductive problems we consider are special cases of the more general problem in Figure 1a where a reasoner is given a partially observed matrix of animals by features then asked to infer the values of the missing entries. This general problem has been previously studied and is addressed by computational models of property induction, categorization, and generalization [1–7]. A challenge faced by all of these models is to capture the background knowledge that guides inductive inferences. Some accounts rely on similarity relationships between animals [6, 8], others rely on causal relationships between features [9, 10], and others incorporate relationships between animals and relationships between features [11]. We will evaluate graphical models that capture both kinds of relationships (Figure 1a), but will focus in particular on relationships between features. Psychologists have previously suggested that humans rely on explicit mental representations of relationships between features [12–16]. Often these representations are described as theories—for example, theories that specify a causal relationship between having wings and ﬂying, or living in the sea and eating ﬁsh. Relationships between features may take several forms: for example, one feature may cause, enable, prevent, be inconsistent with, or be a special case of another feature. For simplicity, we will treat all of these relationships as instances of dependency relationships between features, and will capture them using an undirected graphical model. Previous studies have used graphical models to account for human inferences about features but typically these studies consider toy problems involving a handful of novel features such as “has gene X14” or “has enzyme Y132” [9, 11]. Participants might be told, for example, that gene X14 leads to the production of enzyme Y132, then asked to use this information when reasoning about novel animals. Here we explore whether a graphical model approach can account for inferences 1 (a) slow heavy flies (b) wings hippo 1 1 0 0 rhino 1 1 0 0 sparrow 0 0 1 1 robin 0 0 1 1 new ? ? 1 ? o Figure 1: Inductive reasoning about animals and features. (a) Inferences about the features of a new animal onew that ﬂies may draw on similarity relationships between animals (the new animal is similar to sparrows and robins but not hippos and rhinos), and on dependency relationships between features (ﬂying and having wings are linked). (b) A graph product produced by combining the two graph structures in (a). about familiar features. Working with familiar features raises a methodological challenge since participants have a substantial amount of knowledge about these features and can reason about them in multiple ways. Suppose, for example, that you learn that a novel animal can ﬂy (Figure 1a). To conclude that the animal probably has wings, you might consult a mental representation similar to the graph at the top of Figure 1a that speciﬁes a dependency relationship between ﬂying and having wings. On the other hand, you might reach the same conclusion by thinking about ﬂying creatures that you have previously encountered (e.g. sparrows and robins) and noticing that these creatures have wings. Since the same conclusion can be reached in two different ways, judgments about arguments of this kind provide little evidence about the mental representations involved. The challenge of working with familiar features directly motivates our focus on chimeras. Inferences about chimeras draw on rich background knowledge but require the reasoner to go beyond past experience in a fundamental way. For example, if you learn that an animal ﬂies and has no legs, you cannot make predictions about the animal by thinking of ﬂying, no-legged creatures that you have previously encountered. You may, however, still be able to infer that the novel animal has wings if you understand the relationship between ﬂying and having wings. We propose that graphical models over features can help to explain how humans make inferences of this kind, and evaluate our approach by comparing it to a family of exemplar models. The next section introduces these models, and we then describe two experiments designed to distinguish between the models. 1 Reasoning about objects and features Our models make use of a binary matrix D where the rows {o1 , . . . , o129 } correspond to objects, and the columns {f 1 , . . . , f 56 } correspond to features. A subset of the objects is shown in Figure 2a, and the full set of features is shown in Figure 2b and its caption. Matrix D was extracted from the Leuven natural concept database [17], which includes 129 animals and 757 features in total. We chose a subset of these features that includes a mix of perceptual and behavioral features, and that includes many pairs of features that depend on each other. For example, animals that “live in water” typically “can swim,” and animals that have “no legs” cannot “jump far.” Matrix D can be used to formulate problems where a reasoner observes one or two features of a new object (i.e. animal o130 ) and must make inferences about the remaining features of the animal. The next two sections describe graphical models that can be used to address this problem. The ﬁrst graphical model O captures relationships between objects, and the second model F captures relationships between features. We then discuss how these models can be combined, and introduce a family of exemplar-style models that will be compared with our graphical models. A graphical model over objects Many accounts of inductive reasoning focus on similarity relationships between objects [6, 8]. Here we describe a tree-structured graphical model O that captures these relationships. The tree was constructed from matrix D using average linkage clustering and the Jaccard similarity measure, and part of the resulting structure is shown in Figure 2a. The subtree in Figure 2a includes clusters 2 alligator caiman crocodile monitor lizard dinosaur blindworm boa cobra python snake viper chameleon iguana gecko lizard salamander frog toad tortoise turtle anchovy herring sardine cod sole salmon trout carp pike stickleback eel flatfish ray plaice piranha sperm whale squid swordfish goldfish dolphin orca whale shark bat fox wolf beaver hedgehog hamster squirrel mouse rabbit bison elephant hippopotamus rhinoceros lion tiger polar bear deer dromedary llama giraffe zebra kangaroo monkey cat dog cow horse donkey pig sheep (a) (b) can swim lives in water eats fish eats nuts eats grain eats grass has gills can jump far has two legs has no legs has six legs has four legs can fly can be ridden has sharp teeth nocturnal has wings strong predator can see in dark eats berries lives in the sea lives in the desert crawls lives in the woods has mane lives in trees can climb well lives underground has feathers has scales slow has fur heavy Figure 2: Graph structures used to deﬁne graphical models O and F. (a) A tree that captures similarity relationships between animals. The full tree includes 129 animals, and only part of the tree is shown here. The grey points along the branches indicate locations where a novel animal o130 could be attached to the tree. (b) A network capturing pairwise dependency relationships between features. The edges capture both positive and negative dependencies. All edges in the network are shown, and the network also includes 20 isolated nodes for the following features: is black, is blue, is green, is grey, is pink, is red, is white, is yellow, is a pet, has a beak, stings, stinks, has a long neck, has feelers, sucks blood, lays eggs, makes a web, has a hump, has a trunk, and is cold-blooded. corresponding to amphibians and reptiles, aquatic creatures, and land mammals, and the subtree omitted for space includes clusters for insects and birds. We assume that the features in matrix D (i.e. the columns) are generated independently over O: P (f i |O, π i , λi ). P (D|O, π, λ) = i i i i The distribution P (f |O, π , λ ) is based on the intuition that nearby nodes in O tend to have the same value of f i . Previous researchers [8, 18] have used a directed graphical model where the distribution at the root node is based on the baserate π i , and any other node v with parent u has the following conditional probability distribution: i P (v = 1|u) = π i + (1 − π i )e−λ l , if u = 1 i π i − π i e−λ l , if u = 0 (1) where l is the length of the branch joining node u to node v. The variability parameter λi captures the extent to which feature f i is expected to vary over the tree. Note, for example, that any node v must take the same value as its parent u when λ = 0. To avoid free parameters, the feature baserates π i and variability parameters λi are set to their maximum likelihood values given the observed values of the features {f i } in the data matrix D. The conditional distributions in Equation 1 induce a joint distribution over all of the nodes in graph O, and the distribution P (f i |O, π i , λi ) is computed by marginalizing out the values of the internal nodes. Although we described O as a directed graphical model, the model can be converted into an equivalent undirected model with a potential for each edge in the tree and a potential for the root node. Here we use the undirected version of the model, which is a natural counterpart to the undirected model F described in the next section. The full version of structure O in Figure 2a includes 129 familiar animals, and our task requires inferences about a novel animal o130 that must be slotted into the structure. Let D′ be an expanded version of D that includes a row for o130 , and let O′ be an expanded version of O that includes a node for o130 . The edges in Figure 2a are marked with evenly spaced gray points, and we use a 3 uniform prior P (O′ ) over all trees that can be created by attaching o130 to one of these points. Some of these trees have identical topologies, since some edges in Figure 2a have multiple gray points. Predictions about o130 can be computed using: P (D′ |D) = P (D′ |O′ , D)P (O′ |D) ∝ O′ P (D′ |O′ , D)P (D|O′ )P (O′ ). (2) O′ Equation 2 captures the basic intuition that the distribution of features for o130 is expected to be consistent with the distribution observed for previous animals. For example, if o130 is known to ﬂy then the trees with high posterior probability P (O′ |D) will be those where o130 is near other ﬂying creatures (Figure 1a), and since these creatures have wings Equation 2 predicts that o130 probably also has wings. As this example suggests, model O captures dependency relationships between features implicitly, and therefore stands in contrast to models like F that rely on explicit representations of relationships between features. A graphical model over features Model F is an undirected graphical model deﬁned over features. The graph shown in Figure 2b was created by identifying pairs where one feature depends directly on another. The author and a research assistant both independently identiﬁed candidate sets of pairwise dependencies, and Figure 2b was created by merging these sets and reaching agreement about how to handle any discrepancies. As previous researchers have suggested [13, 15], feature dependencies can capture several kinds of relationships. For example, wings enable ﬂying, living in the sea leads to eating ﬁsh, and having no legs rules out jumping far. We work with an undirected graph because some pairs of features depend on each other but there is no clear direction of causal inﬂuence. For example, there is clearly a dependency relationship between being nocturnal and seeing in the dark, but no obvious sense in which one of these features causes the other. We assume that the rows of the object-feature matrix D are generated independently from an undirected graphical model F deﬁned over the feature structure in Figure 2b: P (oi |F). P (D|F) = i Model F includes potential functions for each node and for each edge in the graph. These potentials were learned from matrix D using the UGM toolbox for undirected graphical models [19]. The learned potentials capture both positive and negative relationships: for example, animals that live in the sea tend to eat ﬁsh, and tend not to eat berries. Some pairs of feature values never occur together in matrix D (there are no creatures that ﬂy but do not have wings). We therefore chose to compute maximum a posteriori values of the potential functions rather than maximum likelihood values, and used a diffuse Gaussian prior with a variance of 100 on the entries in each potential. After learning the potentials for model F, we can make predictions about a new object o130 using the distribution P (o130 |F). For example, if o130 is known to ﬂy (Figure 1a), model F predicts that o130 probably has wings because the learned potentials capture a positive dependency between ﬂying and having wings. Combining object and feature relationships There are two simple ways to combine models O and F in order to develop an approach that incorporates both relationships between features and relationships between objects. The output combination model computes the predictions of both models in isolation, then combines these predictions using a weighted sum. The resulting model is similar to a mixture-of-experts model, and to avoid free parameters we use a mixing weight of 0.5. The structure combination model combines the graph structures used by the two models and relies on a set of potentials deﬁned over the resulting graph product. An example of a graph product is shown in Figure 1b, and the potential functions for this graph are inherited from the component models in the natural way. Kemp et al. [11] use a similar approach to combine a functional causal model with an object model O, but note that our structure combination model uses an undirected model F rather than a functional causal model over features. Both combination models capture the intuition that inductive inferences rely on relationships between features and relationships between objects. The output combination model has the virtue of 4 simplicity, and the structure combination model is appealing because it relies on a single integrated representation that captures both relationships between features and relationships between objects. To preview our results, our data suggest that the combination models perform better overall than either O or F in isolation, and that both combination models perform about equally well. Exemplar models We will compare the family of graphical models already described with a family of exemplar models. The key difference between these model families is that the exemplar models do not rely on explicit representations of relationships between objects and relationships between features. Comparing the model families can therefore help to establish whether human inferences rely on representations of this sort. Consider ﬁrst a problem where a reasoner must predict whether object o130 has feature k after observing that it has feature i. An exemplar model addresses the problem by retrieving all previouslyobserved objects with feature i and computing the proportion that have feature k: P (ok = 1|oi = 1) = |f k & f i | |f i | (3) where |f k | is the number of objects in matrix D that have feature k, and |f k & f i | is the number that have both feature k and feature i. Note that we have streamlined our notation by using ok instead of o130 to refer to the kth feature value for object o130 . k Suppose now that the reasoner observes that object o130 has features i and j. The natural generalization of Equation 3 is: P (ok = 1|oi = 1, oj = 1) = |f k & f i & f j | |f i & f j | (4) Because we focus on chimeras, |f i & f j | = 0 and Equation 4 is not well deﬁned. We therefore evaluate an exemplar model that computes predictions for the two observed features separately then computes the weighted sum of these predictions: P (ok = 1|oi = 1, oj = 1) = wi |f k & f i | |f k & f j | + wj . i| |f |f j | (5) where the weights wi and wj must sum to one. We consider four ways in which the weights could be set. The ﬁrst strategy sets wi = wj = 0.5. The second strategy sets wi ∝ |f i |, and is consistent with an approach where the reasoner retrieves all exemplars in D that are most similar to the novel animal and reports the proportion of these exemplars that have feature k. The third strategy sets wi ∝ |f1i | , and captures the idea that features should be weighted by their distinctiveness [20]. The ﬁnal strategy sets weights according to the coherence of each feature [21]. A feature is coherent if objects with that feature tend to resemble each other overall, and we deﬁne the coherence of feature i as the expected Jaccard similarity between two randomly chosen objects from matrix D that both have feature i. Note that the ﬁnal three strategies are all consistent with previous proposals from the psychological literature, and each one might be expected to perform well. Because exemplar models and prototype models are often compared, it is natural to consider a prototype model [22] as an additional baseline. A standard prototype model would partition the 129 animals into categories and would use summary statistics for these categories to make predictions about the novel animal o130 . We will not evaluate this model because it corresponds to a coarser version of model O, which organizes the animals into a hierarchy of categories. The key characteristic shared by both models is that they explicitly capture relationships between objects but not features. 2 Experiment 1: Chimeras Our ﬁrst experiment explores how people make inferences about chimeras, or novel animals with features that have not previously been observed to co-occur. Inferences about chimeras raise challenges for exemplar models, and therefore help to establish whether humans rely on explicit representations of relationships between features. Each argument can be represented as f i , f j → f k 5 exemplar r = 0.42 7 feature F exemplar (wi = |f i |) (wi = 0.5) r = 0.44 7 object O r = 0.69 7 output combination r = 0.31 7 structure combination r = 0.59 7 r = 0.60 7 5 5 5 5 5 3 3 3 3 3 3 all 5 1 1 0 1 r = 0.06 7 conflict 0.5 1 1 0 0.5 1 r = 0.71 7 1 0 0.5 1 r = −0.02 7 1 0 0.5 1 r = 0.49 7 0 5 5 5 5 3 3 3 3 1 5 3 0.5 r = 0.57 7 5 3 1 0 0.5 1 r = 0.51 7 edge 0.5 r = 0.17 7 1 1 0 0.5 1 r = 0.64 7 1 0 0.5 1 r = 0.83 7 1 0 0.5 1 r = 0.45 7 1 0 0.5 1 r = 0.76 7 0 5 5 5 5 3 3 3 3 1 5 3 0.5 r = 0.79 7 5 3 1 1 0 0.5 1 r = 0.26 7 other 1 0 1 0 0.5 1 r = 0.25 7 1 0 0.5 1 r = 0.19 7 1 0 0.5 1 r = 0.25 7 1 0 0.5 1 r = 0.24 7 0 7 5 5 5 5 5 3 3 3 3 1 5 3 0.5 r = 0.33 3 1 1 0 0.5 1 1 0 0.5 1 1 0 0.5 1 1 0 0.5 1 1 0 0.5 1 0 0.5 1 Figure 3: Argument ratings for Experiment 1 plotted against the predictions of six models. The y-axis in each panel shows human ratings on a seven point scale, and the x-axis shows probabilities according to one of the models. Correlation coefﬁcients are shown for each plot. where f i and f k are the premises (e.g. “has no legs” and “can ﬂy”) and f k is the conclusion (e.g. “has wings”). We are especially interested in conﬂict cases where the premises f i and f j lead to opposite conclusions when taken individually: for example, most animals with no legs do not have wings, but most animals that ﬂy do have wings. Our models that incorporate feature structure F can resolve this conﬂict since F includes a dependency between “wings” and “can ﬂy” but not between “wings” and “has no legs.” Our models that do not include F cannot resolve the conﬂict and predict that humans will be uncertain about whether the novel animal has wings. Materials. The object-feature matrix D includes 447 feature pairs {f i , f j } such that none of the 129 animals has both f i and f j . We selected 40 pairs (see the supporting material) and created 400 arguments in total by choosing 10 conclusion features for each pair. The arguments can be assigned to three categories. Conﬂict cases are arguments f i , f j → f k such that the single-premise arguments f i → f k and f j → f k lead to incompatible predictions. For our purposes, two singlepremise arguments with the same conclusion are deemed incompatible if one leads to a probability greater than 0.9 according to Equation 3, and the other leads to a probability less than 0.1. Edge cases are arguments f i , f j → f k such that the feature network in Figure 2b includes an edge between f k and either f i or f j . Note that some arguments are both conﬂict cases and edge cases. All arguments that do not fall into either one of these categories will be referred to as other cases. The 400 arguments for the experiment include 154 conﬂict cases, 153 edge cases, and 120 other cases. 34 arguments are both conﬂict cases and edge cases. We chose these arguments based on three criteria. First, we avoided premise pairs that did not co-occur in matrix D but that co-occur in familiar animals that do not belong to D. For example, “is pink” and “has wings” do not co-occur in D but “ﬂamingo” is a familiar animal that has both features. Second, we avoided premise pairs that speciﬁed two different numbers of legs—for example, {“has four legs,” “has six legs”}. Finally, we aimed to include roughly equal numbers of conﬂict cases, edge cases, and other cases. Method. 16 undergraduates participated for course credit. The experiment was carried out using a custom-built computer interface, and one argument was presented on screen at a time. Participants 6 rated the probability of the conclusion on seven point scale where the endpoints were labeled “very unlikely” and “very likely.” The ten arguments for each pair of premises were presented in a block, but the order of these blocks and the order of the arguments within these blocks were randomized across participants. Results. Figure 3 shows average human judgments plotted against the predictions of six models. The plots in the ﬁrst row include all 400 arguments in the experiment, and the remaining rows show results for conﬂict cases, edge cases, and other cases. The previous section described four exemplar models, and the two shown in Figure 3 are the best performers overall. Even though the graphical models include more numerical parameters than the exemplar models, recall that these parameters are learned from matrix D rather than ﬁt to the experimental data. Matrix D also serves as the basis for the exemplar models, which means that all of the models can be compared on equal terms. The ﬁrst row of Figure 3 suggests that the three models which include feature structure F perform better than the alternatives. The output combination model is the worst of the three models that incorporate F, and the correlation achieved by this model is signiﬁcantly greater than the correlation achieved by the best exemplar model (p < 0.001, using the Fisher transformation to convert correlation coefﬁcients to z scores). Our data therefore suggest that explicit representations of relationships between features are needed to account for inductive inferences about chimeras. The model that includes the feature structure F alone performs better than the two models that combine F with the object structure O, which may not be surprising since Experiment 1 focuses speciﬁcally on novel animals that do not slot naturally into structure O. Rows two through four suggest that the conﬂict arguments in particular raise challenges for the models which do not include feature structure F. Since these conﬂict cases are arguments f i , f j → f k where f i → f k has strength greater than 0.9 and f j → f k has strength less than 0.1, the ﬁrst exemplar model averages these strengths and assigns an overall strength of around 0.5 to each argument. The second exemplar model is better able to differentiate between the conﬂict arguments, but still performs substantially worse than the three models that include structure F. The exemplar models perform better on the edge arguments, but are outperformed by the models that include F. Finally, all models achieve roughly the same level of performance on the other arguments. Although the feature model F performs best overall, the predictions of this model still leave room for improvement. The two most obvious outliers in the third plot in the top row represent the arguments {is blue, lives in desert → lives in woods} and {is pink, lives in desert → lives in woods}. Our participants sensibly infer that any animal which lives in the desert cannot simultaneously live in the woods. In contrast, the Leuven database indicates that eight of the twelve animals that live in the desert also live in the woods, and the edge in Figure 2b between “lives in the desert” and “lives in the woods” therefore represents a positive dependency relationship according to model F. This discrepancy between model and participants reﬂects the fact that participants made inferences about individual animals but the Leuven database is based on features of animal categories. Note, for example, that any individual animal is unlikely to live in the desert and the woods, but that some animal categories (including snakes, salamanders, and lizards) are found in both environments. 3 Experiment 2: Single-premise arguments Our results so far suggest that inferences about chimeras rely on explicit representations of relationships between features but provide no evidence that relationships between objects are important. It would be a mistake, however, to conclude that relationships between objects play no role in inductive reasoning. Previous studies have used object structures like the example in Figure 2a to account for inferences about novel features [11]—for example, given that alligators have enzyme Y132 in their blood, it seems likely that crocodiles also have this enzyme. Inferences about novel objects can also draw on relationships between objects rather than relationships between features. For example, given that a novel animal has a beak you will probably predict that it has feathers, not because there is any direct dependency between these two features, but because the beaked animals that you know tend to have feathers. Our second experiment explores inferences of this kind. Materials and Method. 32 undergraduates participated for course credit. The task was identical to Experiment 1 with the following exceptions. Each two-premise argument f i , f j → f k from Experiment 1 was converted into two one-premise arguments f i → f k and f j → f k , and these 7 feature F exemplar r = 0.78 7 object O r = 0.54 7 output combination r = 0.75 7 structure combination r = 0.75 7 all 5 5 5 5 5 3 3 3 3 3 1 1 0 edge 0.5 1 r = 0.87 7 1 0 0.5 1 r = 0.87 7 1 0 0.5 1 r = 0.84 7 1 0 0.5 1 r = 0.86 7 0 5 5 5 3 3 3 1 5 3 0.5 r = 0.85 7 5 3 1 1 0 0.5 1 r = 0.79 7 other r = 0.77 7 1 0 0.5 1 r = 0.21 7 1 0 0.5 1 r = 0.74 7 1 0 0.5 1 r = 0.66 7 0 5 5 5 5 3 3 3 3 1 r = 0.73 7 5 0.5 3 1 1 0 0.5 1 1 0 0.5 1 1 0 0.5 1 1 0 0.5 1 0 0.5 1 Figure 4: Argument ratings and model predictions for Experiment 2. one-premise arguments were randomly assigned to two sets. 16 participants rated the 400 arguments in the ﬁrst set, and the other 16 rated the 400 arguments in the second set. Results. Figure 4 shows average human ratings for the 800 arguments plotted against the predictions of ﬁve models. Unlike Figure 3, Figure 4 includes a single exemplar model since there is no need to consider different feature weightings in this case. Unlike Experiment 1, the feature model F performs worse than the other alternatives (p < 0.001 in all cases). Not surprisingly, this model performs relatively well for edge cases f j → f k where f j and f k are linked in Figure 2b, but the ﬁnal row shows that the model performs poorly across the remaining set of arguments. Taken together, Experiments 1 and 2 suggest that relationships between objects and relationships between features are both needed to account for human inferences. Experiment 1 rules out an exemplar approach but models that combine graph structures over objects and features perform relatively well in both experiments. We considered two methods for combining these structures and both performed equally well. Combining the knowledge captured by these structures appears to be important, and future studies can explore in detail how humans achieve this combination. 4 Conclusion This paper proposed that graphical models are useful for capturing knowledge about animals and their features and showed that a graphical model over features can account for human inferences about chimeras. A family of exemplar models and a graphical model deﬁned over objects were unable to account for our data, which suggests that humans rely on mental representations that explicitly capture dependency relationships between features. Psychologists have previously used graphical models to capture relationships between features, but our work is the ﬁrst to focus on chimeras and to explore models deﬁned over a large set of familiar features. Although a simple undirected model accounted relatively well for our data, this model is only a starting point. The model incorporates dependency relationships between features, but people know about many speciﬁc kinds of dependencies, including cases where one feature causes, enables, prevents, or is inconsistent with another. An undirected graph with only one class of edges cannot capture this knowledge in full, and richer representations will ultimately be needed in order to provide a more complete account of human reasoning. Acknowledgments I thank Madeleine Clute for assisting with this research. This work was supported in part by the Pittsburgh Life Sciences Greenhouse Opportunity Fund and by NSF grant CDI-0835797. 8 References [1] R. N. Shepard. Towards a universal law of generalization for psychological science. Science, 237:1317– 1323, 1987. [2] J. R. Anderson. The adaptive nature of human categorization. Psychological Review, 98(3):409–429, 1991. [3] E. Heit. A Bayesian analysis of some forms of inductive reasoning. In M. Oaksford and N. Chater, editors, Rational models of cognition, pages 248–274. Oxford University Press, Oxford, 1998. [4] J. B. Tenenbaum and T. L. Grifﬁths. Generalization, similarity, and Bayesian inference. Behavioral and Brain Sciences, 24:629–641, 2001. [5] C. Kemp and J. B. Tenenbaum. Structured statistical models of inductive reasoning. Psychological Review, 116(1):20–58, 2009. [6] D. N. Osherson, E. E. Smith, O. Wilkie, A. Lopez, and E. Shaﬁr. Category-based induction. Psychological Review, 97(2):185–200, 1990. [7] D. J. Navarro. Learning the context of a category. In J. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R.S. Zemel, and A. Culotta, editors, Advances in Neural Information Processing Systems 23, pages 1795–1803. 2010. [8] C. Kemp, T. L. Grifﬁths, S. Stromsten, and J. B. Tenenbaum. Semi-supervised learning with trees. In Advances in Neural Information Processing Systems 16, pages 257–264. MIT Press, Cambridge, MA, 2004. [9] B. Rehder. A causal-model theory of conceptual representation and categorization. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29:1141–1159, 2003. [10] B. Rehder and R. Burnett. Feature inference and the causal structure of categories. Cognitive Psychology, 50:264–314, 2005. [11] C. Kemp, P. Shafto, and J. B. Tenenbaum. An integrated account of generalization across objects and features. Cognitive Psychology, in press. [12] S. E. Barrett, H. Abdi, G. L. Murphy, and J. McCarthy Gallagher. Theory-based correlations and their role in children’s concepts. Child Development, 64:1595–1616, 1993. [13] S. A. Sloman, B. C. Love, and W. Ahn. Feature centrality and conceptual coherence. Cognitive Science, 22(2):189–228, 1998. [14] D. Yarlett and M. Ramscar. A quantitative model of counterfactual reasoning. In T. G. Dietterich, S. Becker, and Z. Ghahramani, editors, Advances in Neural Information Processing Systems 14, pages 123–130. MIT Press, Cambridge, MA, 2002. [15] W. Ahn, J. K. Marsh, C. C. Luhmann, and K. Lee. Effect of theory-based feature correlations on typicality judgments. Memory and Cognition, 30(1):107–118, 2002. [16] D. C. Meehan C. McNorgan, R. A. Kotack and K. McRae. Feature-feature causal relations and statistical co-occurrences in object concepts. Memory and Cognition, 35(3):418–431, 2007. [17] S. De Deyne, S. Verheyen, E. Ameel, W. Vanpaemel, M. J. Dry, W. Voorspoels, and G. Storms. Exemplar by feature applicability matrices and other Dutch normative data for semantic concepts. Behavior Research Methods, 40(4):1030–1048, 2008. [18] J. P. Huelsenbeck and F. Ronquist. MRBAYES: Bayesian inference of phylogenetic trees. Bioinformatics, 17(8):754–755, 2001. [19] M. Schmidt. UGM: A Matlab toolbox for probabilistic undirected graphical models. 2007. Available at http://people.cs.ubc.ca/∼schmidtm/Software/UGM.html. [20] L. J. Nelson and D. T. Miller. The distinctiveness effect in social categorization: you are what makes you unusual. Psychological Science, 6:246–249, 1995. [21] A. L. Patalano, S. Chin-Parker, and B. H. Ross. The importance of being coherent: category coherence, cross-classiﬁcation and reasoning. Journal of memory and language, 54:407–424, 2006. [22] S. K. Reed. Pattern recognition and categorization. Cognitive Psychology, 3:393–407, 1972. 9

5 0.10680538 190 nips-2011-Nonlinear Inverse Reinforcement Learning with Gaussian Processes

Author: Sergey Levine, Zoran Popovic, Vladlen Koltun

Abstract: We present a probabilistic algorithm for nonlinear inverse reinforcement learning. The goal of inverse reinforcement learning is to learn the reward function in a Markov decision process from expert demonstrations. While most prior inverse reinforcement learning algorithms represent the reward as a linear combination of a set of features, we use Gaussian processes to learn the reward as a nonlinear function, while also determining the relevance of each feature to the expert’s policy. Our probabilistic algorithm allows complex behaviors to be captured from suboptimal stochastic demonstrations, while automatically balancing the simplicity of the learned reward structure against its consistency with the observed actions. 1

6 0.060997847 98 nips-2011-From Bandits to Experts: On the Value of Side-Observations

7 0.058329135 215 nips-2011-Policy Gradient Coagent Networks

8 0.057235155 245 nips-2011-Selecting the State-Representation in Reinforcement Learning

9 0.05693201 219 nips-2011-Predicting response time and error rates in visual search

10 0.05469517 174 nips-2011-Monte Carlo Value Iteration with Macro-Actions

11 0.050281428 16 nips-2011-A reinterpretation of the policy oscillation phenomenon in approximate policy iteration

12 0.049324591 10 nips-2011-A Non-Parametric Approach to Dynamic Programming

13 0.047655076 210 nips-2011-PAC-Bayesian Analysis of Contextual Bandits

14 0.045461256 18 nips-2011-Action-Gap Phenomenon in Reinforcement Learning

15 0.044891689 249 nips-2011-Sequence learning with hidden units in spiking neural networks

16 0.044462547 206 nips-2011-Optimal Reinforcement Learning for Gaussian Systems

17 0.041337978 128 nips-2011-Improved Algorithms for Linear Stochastic Bandits

18 0.041000552 184 nips-2011-Neuronal Adaptation for Sampling-Based Probabilistic Inference in Perceptual Bistability

19 0.04014821 75 nips-2011-Dynamical segmentation of single trials from population neural data

20 0.037035964 300 nips-2011-Variance Reduction in Monte-Carlo Tree Search

similar papers computed by lsi model

lsi for this paper:

topicId topicWeight

[(0, 0.111), (1, -0.074), (2, 0.092), (3, 0.119), (4, -0.076), (5, 0.01), (6, 0.022), (7, -0.009), (8, 0.006), (9, 0.002), (10, -0.002), (11, -0.003), (12, 0.041), (13, 0.038), (14, 0.085), (15, -0.025), (16, 0.049), (17, 0.035), (18, 0.049), (19, -0.063), (20, -0.046), (21, 0.029), (22, -0.098), (23, 0.056), (24, 0.046), (25, 0.05), (26, 0.026), (27, -0.142), (28, 0.009), (29, 0.058), (30, 0.021), (31, -0.027), (32, -0.056), (33, -0.075), (34, -0.021), (35, -0.023), (36, -0.049), (37, 0.007), (38, 0.05), (39, 0.038), (40, 0.09), (41, -0.162), (42, -0.078), (43, -0.106), (44, 0.042), (45, -0.147), (46, 0.031), (47, 0.035), (48, -0.059), (49, -0.172)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.95837235 11 nips-2011-A Reinforcement Learning Theory for Homeostatic Regulation

Author: Mehdi Keramati, Boris S. Gutkin

2 0.67288208 190 nips-2011-Nonlinear Inverse Reinforcement Learning with Gaussian Processes

Author: Sergey Levine, Zoran Popovic, Vladlen Koltun

3 0.67126375 163 nips-2011-MAP Inference for Bayesian Inverse Reinforcement Learning

Author: Jaedeug Choi, Kee-eung Kim

4 0.66313589 88 nips-2011-Environmental statistics and the trade-off between model-based and TD learning in humans

Author: Dylan A. Simon, Nathaniel D. Daw

5 0.48367915 130 nips-2011-Inductive reasoning about chimeric creatures

Author: Charles Kemp

6 0.39760324 98 nips-2011-From Bandits to Experts: On the Value of Side-Observations

7 0.36849278 237 nips-2011-Reinforcement Learning using Kernel-Based Stochastic Factorization

8 0.36457539 10 nips-2011-A Non-Parametric Approach to Dynamic Programming

9 0.3639003 15 nips-2011-A rational model of causal inference with continuous causes

10 0.34069079 219 nips-2011-Predicting response time and error rates in visual search

11 0.33904892 38 nips-2011-Anatomically Constrained Decoding of Finger Flexion from Electrocorticographic Signals

12 0.32079083 90 nips-2011-Evaluating the inverse decision-making approach to preference learning

13 0.31306177 48 nips-2011-Blending Autonomous Exploration and Apprenticeship Learning

14 0.30764687 224 nips-2011-Probabilistic Modeling of Dependencies Among Visual Short-Term Memory Representations

15 0.3035455 174 nips-2011-Monte Carlo Value Iteration with Macro-Actions

16 0.28311765 85 nips-2011-Emergence of Multiplication in a Biophysical Model of a Wide-Field Visual Neuron for Computing Object Approaches: Dynamics, Peaks, & Fits

17 0.27636597 122 nips-2011-How Do Humans Teach: On Curriculum Learning and Teaching Dimension

18 0.27568313 194 nips-2011-On Causal Discovery with Cyclic Additive Noise Models

19 0.26958117 245 nips-2011-Selecting the State-Representation in Reinforcement Learning

20 0.26746601 27 nips-2011-Advice Refinement in Knowledge-Based SVMs

similar papers computed by lda model

lda for this paper:

topicId topicWeight

[(0, 0.015), (4, 0.036), (20, 0.02), (26, 0.02), (31, 0.154), (33, 0.017), (34, 0.361), (43, 0.038), (45, 0.071), (57, 0.035), (65, 0.016), (74, 0.042), (83, 0.049), (99, 0.033)]

similar papers list:

simIndex simValue paperId paperTitle

same-paper 1 0.80191052 11 nips-2011-A Reinforcement Learning Theory for Homeostatic Regulation

Author: Mehdi Keramati, Boris S. Gutkin

2 0.69909167 158 nips-2011-Learning unbelievable probabilities

Author: Xaq Pitkow, Yashar Ahmadian, Ken D. Miller

Abstract: Loopy belief propagation performs approximate inference on graphical models with loops. One might hope to compensate for the approximation by adjusting model parameters. Learning algorithms for this purpose have been explored previously, and the claim has been made that every set of locally consistent marginals can arise from belief propagation run on a graphical model. On the contrary, here we show that many probability distributions have marginals that cannot be reached by belief propagation using any set of model parameters or any learning algorithm. We call such marginals ‘unbelievable.’ This problem occurs whenever the Hessian of the Bethe free energy is not positive-deﬁnite at the target marginals. All learning algorithms for belief propagation necessarily fail in these cases, producing beliefs or sets of beliefs that may even be worse than the pre-learning approximation. We then show that averaging inaccurate beliefs, each obtained from belief propagation using model parameters perturbed about some learned mean values, can achieve the unbelievable marginals. 1

3 0.57373148 257 nips-2011-SpaRCS: Recovering low-rank and sparse matrices from compressive measurements

Author: Andrew E. Waters, Aswin C. Sankaranarayanan, Richard Baraniuk

Abstract: We consider the problem of recovering a matrix M that is the sum of a low-rank matrix L and a sparse matrix S from a small set of linear measurements of the form y = A(M) = A(L + S). This model subsumes three important classes of signal recovery problems: compressive sensing, afﬁne rank minimization, and robust principal component analysis. We propose a natural optimization problem for signal recovery under this model and develop a new greedy algorithm called SpaRCS to solve it. Empirically, SpaRCS inherits a number of desirable properties from the state-of-the-art CoSaMP and ADMiRA algorithms, including exponential convergence and efﬁcient implementation. Simulation results with video compressive sensing, hyperspectral imaging, and robust matrix completion data sets demonstrate both the accuracy and efﬁcacy of the algorithm. 1

4 0.54651028 140 nips-2011-Kernel Embeddings of Latent Tree Graphical Models

Author: Le Song, Eric P. Xing, Ankur P. Parikh

Abstract: Latent tree graphical models are natural tools for expressing long range and hierarchical dependencies among many variables which are common in computer vision, bioinformatics and natural language processing problems. However, existing models are largely restricted to discrete and Gaussian variables due to computational constraints; furthermore, algorithms for estimating the latent tree structure and learning the model parameters are largely restricted to heuristic local search. We present a method based on kernel embeddings of distributions for latent tree graphical models with continuous and non-Gaussian variables. Our method can recover the latent tree structures with provable guarantees and perform local-minimum free parameter learning and efﬁcient inference. Experiments on simulated and real data show the advantage of our proposed approach. 1

5 0.50269806 75 nips-2011-Dynamical segmentation of single trials from population neural data

Author: Biljana Petreska, Byron M. Yu, John P. Cunningham, Gopal Santhanam, Stephen I. Ryu, Krishna V. Shenoy, Maneesh Sahani

Abstract: Simultaneous recordings of many neurons embedded within a recurrentlyconnected cortical network may provide concurrent views into the dynamical processes of that network, and thus its computational function. In principle, these dynamics might be identiﬁed by purely unsupervised, statistical means. Here, we show that a Hidden Switching Linear Dynamical Systems (HSLDS) model— in which multiple linear dynamical laws approximate a nonlinear and potentially non-stationary dynamical process—is able to distinguish different dynamical regimes within single-trial motor cortical activity associated with the preparation and initiation of hand movements. The regimes are identiﬁed without reference to behavioural or experimental epochs, but nonetheless transitions between them correlate strongly with external events whose timing may vary from trial to trial. The HSLDS model also performs better than recent comparable models in predicting the ﬁring rate of an isolated neuron based on the ﬁring rates of others, suggesting that it captures more of the “shared variance” of the data. Thus, the method is able to trace the dynamical processes underlying the coordinated evolution of network activity in a way that appears to reﬂect its computational role. 1

6 0.49682403 157 nips-2011-Learning to Search Efficiently in High Dimensions

7 0.46736044 249 nips-2011-Sequence learning with hidden units in spiking neural networks

8 0.4645223 23 nips-2011-Active dendrites: adaptation to spike-based communication

9 0.46173733 137 nips-2011-Iterative Learning for Reliable Crowdsourcing Systems

10 0.46118206 241 nips-2011-Scalable Training of Mixture Models via Coresets

11 0.4606275 240 nips-2011-Robust Multi-Class Gaussian Process Classification

12 0.45956123 229 nips-2011-Query-Aware MCMC

13 0.45271951 243 nips-2011-Select and Sample - A Model of Efficient Neural Inference and Learning

14 0.45231935 225 nips-2011-Probabilistic amplitude and frequency demodulation

15 0.45051873 292 nips-2011-Two is better than one: distinct roles for familiarity and recollection in retrieving palimpsest memories

16 0.45036352 88 nips-2011-Environmental statistics and the trade-off between model-based and TD learning in humans

17 0.44688788 221 nips-2011-Priors over Recurrent Continuous Time Processes

18 0.44478819 94 nips-2011-Facial Expression Transfer with Input-Output Temporal Restricted Boltzmann Machines

19 0.44279423 86 nips-2011-Empirical models of spiking in neural populations

20 0.44250134 38 nips-2011-Anatomically Constrained Decoding of Finger Flexion from Electrocorticographic Signals