用于分配加固学习的对手纹状体电路

扩展数据图1附加行为分析。一个

，在面板中进行行为分类分析的示意图be。对应于同一分布的气味被视为同一类。这是针对固定与可变odour分类的情况进行说明的，背景阴影（黄色与灰色）指示分类器的目标。b，行为分类的示意图。在每个验证折叠上，搅拌，跑步，瞳孔区域，舔和训练集中的前50个面部运动能量PC被z键s缩放，然后通过线性内核传递到支持向量分类器（SVC），这可以预测关联的分配。c，正交分析的示意图。SVC学到的权重定义了最能最能分离分布的超平面的矢量正交。可以通过将每个试验的平均奖励（价值方向）对相应的行为回归变量进行回归来定义单独的向量。尽管SVC超平面一次仅考虑四个气味，但回归方向考虑了所有六种气味。d，分类器重量向量与值方向之间的余弦相似性。固定试验和可变试验之间的行为上的任何差异都是正交的价值（相对于0：0的机会水平：p<<0.001，没有固定，p<<0.001，没有任何变量，p=固定与变量的0.154）。e，在示例会话中对应于面对运动能量PC的空间掩模，并用方差排序。连续的PC强调了鼠标搅拌，嗅探和舔行为的更精细方面。f，晚期痕量和基线之间的舔率差异（在气味发作前1）对所有试验类型都显着，包括两种无异味的基线降低（所有）pS <0.001）。g，根据先前的气味试验是否导致2或6â¼l的奖励，可变气味的预期舔率没有差异（p= 0.179）。h，经过培训的线性分类器预测给定气味的先前变量试验中提供的奖励量的偶然准确性为50％（p= 0.326）。扩展数据图2值，RPE，气味和风险编码在整个纹状体上。

一个

，串行冠状切片显示探针插入的记录位点（白色虚线），记录在Allen Common Coartial坐标框架上。b，，，，顶部，热图显示了每个神经元的每种气味的平均Z得分发射速率。按照峰值活性的时间进行分类，当时在2个可变的2个气味试验中平均，然后按照相同的顺序绘制了以试验类型分组的其余试验。第七个也是最后一次试验类型对应于意外的奖励，而这些奖励之前没有气味。底部，所有神经元的平均Z得分射击率。c，与平均奖励显着相关的神经元的一部分，在非重叠的250（250 MS时箱）中分别计算出来。每只小鼠以不同的颜色显示，平均±95％c.i。跨黑色显示的小鼠。虚线是在将气味和分布之间的映射改组后的小鼠的平均值，从而考虑了纯气味编码。d，在痕量后期，重要细胞的平均百分比（p<<0.001）。e，，，，左边，交叉验证r2^{预测每个试验中的平均奖励是纹状体子区域的函数，在非重叠的250 ms时箱中分别计算出来。}为了确保各个区域之间的公平比较，对于每只动物，我们通过反复采样而无需替换神经亚群，直到剩下的神经元少于40个神经元，而无需替换神经亚群来产生多个40个神经元的伪群。在给定区域中，神经元少于40的动物被排除在外。线显示每个子区域跨小鼠的平均值。正确的， 平均的r²在痕量后期。较小的点显示，该区域中至少有40个神经元的每只小鼠跨伪群的平均值。f，与c，除了显示与奖励预测误差（RPE）显着相关的神经元的比例，该神经元被定义为实际奖励和预期奖励之间的差异。g，与d，除了显示结果期间重要细胞的平均百分比，奖励交付后0（0p<<0.001）。h，将与平均值和RPE显着相关的每只小鼠中细胞的实际分数与平均分数和RPE编码细胞的单个分数的乘积进行了比较（假定独立性的预测分数；p<<0.001）。我，，，，左边，在多项式逻辑回归分类器解码气味身份的时间的时间内解码精度（虚线= 1/6的机会水平）。正确的，量化气味分类准确性（气味期）（p相对于机会水平，<0.001）。j，在气味期间的气味解码的混淆矩阵显示出所有气味的高解码精度，对于具有相同平均值的气味，相对较高的混淆性。k，跨阶时解码显示，气味解码在时间范围内是稳定的，允许训练有素的分类器，例如在痕量后期的活动中，将概念概述到偶然时期的气味时期，反之亦然（所有人都p相对于1/6的机会水平，S <0.001）。l，伪造的气味跨区域解码（见 方法题为跨区域，半球和基因型的比较。OT，嗅觉结节；副总裁，腹侧颗粒；MacBSH，伏伏壳中部核；LACBSH，外侧核伏壳；核心，伏隔核；VM，腹侧纹状体；VLS，腹外侧纹状体；DMS，背侧纹状体；DLS，背外侧纹状体（n= Macbsh的1鼠p= VM的0.006，所有其他pS <0.001）。m，与c，除了显示与每次bin的平均奖励编码的贡献后，除了与方差显着相关的神经元的分数外。n，在痕量后期晚期，显着残留方差单元的平均百分比为较少的比单独的气味编码所预测的p<<0.001）。o，明显少于偶然的正面和负差异的神经元对剩余方差进行编码（正和负面）pS <0.001）。p-r，与M-O，但对于有条件的价值（CVAR），这是一种用于金融和增强学习中的常见风险措施126，，，，^{127，，，，128，定义为较低内的期望值±} - 概率分布的定量。对于我们的分布，这将等同于均值±> 0.5，相当于最低值±<0.5，仅针对变量分布而有所不同，其中是2。后者是我们在回归平均编码后在这里绘制的。同样，残留的CVAR细胞比单独的气味编码所预期的要少（p<<0.001），对于阳性和负编码细胞都是如此（均为pS <0.001）。扩展数据图3分布编码是强大的，正交对价值，并且在时间上保持一致。一个

，成对解码分析的示意图。

对线性SVC进行了单个固定和可变气味的培训，一次是两个。This resulted in six possible dichotomies, four of which encompassed one Fixed and one Variable odour (green arrows; “Across distribution”) and two of which compared odours cuing the same exact distribution (orange arrows; “Within distribution”).b, Pairwise decoding during the late trace period was significantly better for across- than within-distribution pairs, consistent with distributional but not traditional RL (p = 0.001).c, Schematic of congruency analysis, which considered all four Fixed and Variable odours simultaneously. In the Congruent grouping, both Fixed odours were assigned to one class (yellow background) and both Variable odours were assigned to the other class (grey background), just as was done for behavioral decoding. By contrast, in the Incongruent groupings, class assignments cut across Fixed and Variable distributions.d, Classifier accuracy in the late trace period was higher for Congruent than Incongruent pairs, again consistent with distributional but not traditional RL (Congruent:p = 0.028 vs. Incongruent 1,p < 0.001 vs. Incongruent 2).e, Schematic illustrating the classifier weight vector (normal to the separating hyperplane for across- or within-distribution classifications) and the regression weight vector (for Value or Variance).f, Quantification of cosine similarity between the classifier weight vector and the Value direction shows that the vectors are not significantly different from orthogonal (CCGP:p = 0.071 cosine similarity relative to chance value of 0; Pairwise:p = 0.797 Across- vs. Within-distribution absolute cosine similarity; Congruency:p = 0.493 Across- vs. Within-distribution absolute cosine similarity).g, Same asf, but for Variance rather than Value direction (p < 0.001 for all comparisons).h-j, Cross-temporal decoding for the pairwise, congruency, and CCGP analyses. Distributional RL is favored during every time period between odour onset and reward delivery, and decoders trained during one period almost always generalize to other time periods.Extended Data Fig. 4 A distribution-coding subpopulation is over-represented in the lAcbSh and permits ANN-based distribution decoding.一个, Pseudo-population CCGP across subregions (relative to chance level of 0.5:p

 = 0.059, 0.473, 0.044, 0.017, 0.088, 0.346, 0.257, 0.407, and 0.133 for OT, VP, mAcbSh, lAcbSh, core, VMS, VLS, DMS, and DLS, respectively. Same order applies to all statistics in this figure). Pseudo-populations were constructed as in Extended Data Fig.

2l。b, Pseudo-population pairwise decoding across subregions (Across- vs. Within-distribution:p = 0.861, 0.344, 0.883, 0.010, 0.409, 0.040, 0.882, 0.482, 0.106).c, Pseudo-population congruency analysis across subregions (Congruent vs. Incongruent 1:p = 0.097, 0.817, 0.744, 0.007, 0.832, 0.047, 0.523, 0.138, 0.523; Congruent vs. Incongruent 2:p = 0.306, 0.760, 0.815, 0.010, 0.473, 0.177, 0.316, 0.486, 0.985).d, Parallelism score across subregions (relative to chance level of 0:p = 0.300, 0.878, 1.00, 0.001, 0.229, 0.243, 0.273, 0.615, 0.764).e，，，，左边, fraction of neurons with classifier coefficients above the percentile cutoff for all three (CCGP, pairwise, and congruency) analyses. Horizontal dotted line indicates level at which 2.5% of null coefficients fell above the cutoff; this was the 73rd percentile (vertical dotted line), and retained 11.43% of neurons.正确的, ratio of data to null coefficients falling above the cutoff (log scale).f, Fraction of distribution-coding cells in each subregion. This fraction is significantly higher in the lAcbSh than in more dorsal subregions (relative to lAcbSh:p = 0.339, 0.285, 0.473, 0.274, 0.071, 0.038, 0.001 for OT, VP, mAcbSh, core, VMS, VLS, and DLS, respectively;p < 0.001 for DMS).g, ANN schematic. Single-trial spike counts from the distribution-coding subpopulation一个were linearly mapped into 16 dimensions by the trainable matrixwand then fed through the network (see 方法）。After a final layer, a softmax function transformed activations into a properly-normalized probability distribution, whose 1-Wasserstein distance to ground truth was minimized with stochastic gradient descent.h, Example decoded distributions from the test set, shown as line plots to distinguish individual pseudo-trials.我, Wasserstein distance relative to reference for the ANN trained on all six trial types, with and without shuffling odour-distribution mappings (p < 0.001 ordered vs. shuffled;p < 0.001 ordered relative to chance value of 1;p = 0.350 shuffled relative to chance value of 1).j, Same as我, but for ANN trained on only the rewarded odours, which shared the same mean (p < 0.001 ordered vs. shuffled, ordered relative to chance value of 1, and shuffled relative to chance value of 1).k, Schematic depicting setup for transfer analysis.Four trial types, including both Nothing odours, were used for training (green background), and the other two were used for testing (orange background).Matched pairings veridically assigned odours to distributions, while mismatched pairings used either only Fixed or only Variable odours for training while assigning one member per training pair and one member per testing pair to the opposite distribution (indicated by the exclamation mark).There were four possible ways to draw the matched dichotomies, all of which are shown (rows).For the mismatched dichotomies, the distributions (Fixed or Variable) could be arbitrarily assigned to both pairs of red and blue odors, and then either red or blue could be assigned to the training versus test set, so only four of the eight total possibilities are显示。l, Wasserstein distance relative to reference for standard (mean ± s.e.m. = 0.128 ± 0.019), matched (0.217 ± 0.032), and mismatched (1.028 ± 0.123) settings. Standard is identical to analysis shown inc, except that for this decoder, neurons from all mice were pooled. Matched transfer yields distributions that are nearly as accurate as training with all six trial types (p < 0.001 for matched vs. mismatched and standard vs. mismatched, Student’st-test for independent samples;p = 0.043 for standard vs. matched, Student’st-test for independent samples;p < 0.001 for standard and matched relative to chance value of 1, one-sample Student’st-测试;p = 0.836 for mismatched relative to chance value of 1, one-sample Student’st-测试）。Extended Data Fig. 5 A generalized linear model (GLM) to examine trial history, reward, reward prediction, and motor encoding in the striatum.一个, Schematic illustrating the design of the GLM (see 方法）。Briefly, trial-length regressors (time in trial and trial history) were broken up into 7 raised cosine basis functions tiling the 6 seconds of each (odour-cued) trial.Reward, reward prediction, and sensory regressors were time-locked to reward or odour onset and then convolved with a logarithmically-scaled raised cosine basis112。Licking, whisking, and running regressors were convolved with the same basis in both the forward and reverse directions.Pupil area and face motion SVDs from Facemap were input directly to the model without convolving.

The Poisson GLM computes the sum of the regressors weighted by their fitted coefficients, passes this through an exponential nonlinearity, and uses this rate to predict spike counts in 20 ms bins.

b，，，，顶部, example regressor matrix for 10 test trials. Each row corresponds to a different predictor, binned on the left by regressor type (rectangles) and group (colour). Rectangles on top demarcate different trials, coloured by trial type.^中间, empirical spike counts in each bin for an example neuron.底部, smoothed empirical firing rate (black) and model prediction (pink) for the trials shown. Deviance statistics in every panel of this figure rely on a held-out test set (never used during cross-validation), after zeroing out the contribution of electrode drift.c, Histogram of fraction deviance explained for all neurons.d, Fraction deviance explained as a function of striatal subregion (relative to DLS:p < 0.001 for OT, VP, lAcbSh, and core;p = 0.490, 0.608, 0.054 for VMS, VLS, and DMS, respectively). For these analyses, mAcbSh was omitted due to lack of neurons/animals.e, Difference in fraction deviance explained between the full model and reduced models in which trial history (上排）， 报酬 （第二行), sensory and reward-prediction (third row), or motor (bottom row) regressors were excluded before re-fitting.f, Kernel strength (see 方法) of trial history (顶部）， 报酬 （第二) expectile (第三), and motor (底部) regressors.g，如e, but showing the difference in fraction deviance explained as a function of striatal subregion. (History, relative to DLS:p = 0.124 for DMS;p < 0.001 for all other subregions; Reward, relative to DLS:p = 0.009, 0.141, and 0.441 for OT, VP, and DMS, respectively;p < 0.001 for all other subregions; Expectiles, relative to DLS:p = 0.234 for DMS;p < 0.001 for all other subregions; Motor, relative to DLS:p < 0.001 for all subregions).h，如f, but showing the kernel strength computed on the full model as a function of striatal subregion. (History, relative to DLS:p < 0.001 for OT, VP, and VLS;p = 0.042, 0.288, 0.023, and 0.926 for lAcbSh, core, VMS, and DMS, respectively; Reward, relative to DLS: 0.148, 0.004, 0.172 for VP, core, and DMS;p < 0.001 for all other subregions; Expectiles, relative to DLS:p < 0.001 for OT, VP, lAcbSh, VMS, and VLS;p = 0.285 and 0.014 for core and DMS, respectively; Motor, relative to DLS:p = 0.004 for DMS;p < 0.001 for all other subregions).我, Pearson correlation (across-neurons, within-sessions) of difference in deviance explained between reduced models. Holding out trial history, reward, or expectiles tends to similarly affect deviance for a given neuron, while being uncorrelated with motor behavior. Small dots, individual sessions; medium dots, mean across sessions within animals; large dots, mean ± 95% c.i. across mice. (Drop History vs. Drop Reward, Drop History vs. Drop Expectiles, and Drop Reward vs. Drop Expectiles,p < 0.001 for all subregions; Drop Motor vs. Drop History,p = 0.644, 0.479, 0.993, 0.428, 0.133, 0.148, 0.674, 0.986 for OT, VP, lAcbSh, core, VMS, VLS, DMS, and DLS respectively; Drop Motor vs. Drop Reward,p = 0.626, 0.981, 0.134, 0.596, 0.473, 0.028, 0.745, 0.498; Drop Motor vs. Drop Expectiles,p = 0.331, 0.816, 0.796, 0.681, 0.193, 0.603, 0.148, 0.554).Extended Data Fig. 6 Striatal activity patterns are inconsistent with sampling-based codes.一个, Illustration of how the mean-matched Fano factor was computed115。The mean and variance (across trials) of the spike count for a single neuron contributed one data point to the scatter plot.Grey dots depict all neurons from an example session, time bin (here, centered 200 ms after odour onset), and odour (here, Variable 2).The grey line is the regression fit to all data, constrained to pass through zero and weighted according to the estimated s.e.m.of each variance measurement.Black dots are the data points preserved by mean matching at each time point, to eliminate the possibility that differences across time are driven by differences in firing rates, which could in principle violate the Poisson assumption.This transforms the distribution of mean counts from the grey to the black distribution.The regression slope for the mean matched data is plotted as the black line.Finally, the Poisson expectation of equal mean and variance is plotted in orange, with a slope of one.This procedure was performed independently on each session, time bin, and trial type.b, Time course of the computed mean-matched Fano factor (±95% c.i.) for the example session shown in一个。That is, the slope of black line in一个

is the height of the light blue, Variable 2 line inb

200 ms after CS onset.c^{, Quantification of mean matched Fano factor across second-long time periods. Consistent with cortical observations115}, we see a quenching of variability upon CS onset (baseline:p = 0.002, 0.001, <0.001, <0.001 relative to odour, early trace, late trace, and outcome periods), and another one upon reward delivery (reward:p < 0.001, = 0.002, 0.006, 0.053 for baseline, odour, early, and late trace periods).d, Quantification of mean matched Fano factor across trial types, shown separately for each time period. In general, there is no tendency for Variable odours to elicit strong and sustained increases in variability, as would be predicted by sampling-based codes129(baseline, odour, early and late trace: allp’s > 0.05, except Nothing 1 vs. Variable 1 for odour:p^{ = 0.032 uncorrected). However, reward delivery specifically drives yet another decrease in variability during the outcome period (Nothing 1:}p = 0.570 for Nothing 2;p < 0.001 for Fixed odours;p = 0.002 for Variable odours).Extended Data Fig. 7 Additional detail for distributional model comparisons.一个, Schematic showing converged expectile code for each distribution (Nothing, Fixed, and Variable) learned by EDRL, as in Fig.2d。^{The activation of each value predictor is shown as a function of}Ï„, the level of pessimism or optimism. Together, they encompass the complete reward distribution.b, Same as一个, but for quantiles rather than expectiles.c, Same asb, but for a reflected quantile code in which pessimistic (D2, green) neurons correlate negatively withv我

（灰色的）。

Optimistic (D1, yellow) neurons are identical tov我, as in REDRL.d, Same as一个, but showing the converged value predictors for the Distributed Actor Uncertainty model123。In it, D1 and D2 MSNs learn exclusively from positive and negative RPEs, respectively, such that their difference at each level ofÏ„(grey dots) approximates each expectile, and their sum relates to the spread of the distribution. This drives maximal activity in response to Variable odours, which is why they separate out most clearly along PC 1.e, Same asd, but for a reduced version in which only a single pair of value predictors are learned with balanced positive and negative learning rates66（（Ï„_{ = 0.5).}f, Same as_一个, but for a categorical code in which distributions are encoded as a histogram33。Each neuron is imagined to correspond to a single reward bin, with its firing rate proportional to the height of that bin.g^{, Same as}f, but for a Laplace code40。In the limit of infinitely steep reward sensitivities for the teaching signal, these value predictors converge to the probability that the reward delivered exceeds some threshold reward amount, the “exceedance probability”.This is simply 1 minus the CDF of the probability distribution in question.Neural activities are taken to be proportional to this 1 – CDF value.h^{, Same as}g, but for a population of neurons that flips the encoding, and so is directly proportional to the CDF.i-k, Qualitative features of each code ina–hplus random noise. REDRL predictions are included in the box on the last line, for comparison.我^{, PCA projection for each code. Only quantile-like codes give rise to the pattern observed in the data.}j, Hypothetical activity in response to each distribution, averaged separately over optimistic (blue) and pessimistic (purple) predictors for each code type. Only the reflected codes and AU model predict a noticeable uptick in Variable relative to Fixed odours.k, Percentage of simulated predictors that significantly correlate with mean reward either positively (blue) or negatively (purple) for each code type. Only the reflected and categorical codes have a substantial fraction of both types of cells. In practice the positive-coding predictors are optimistic and the negative-coding predictors are pessimistic.l^{, A hypothetical “distributional” code in which each neuron’s firing rate linearly correlates with either reward mean (}左边) or variance (正确的）。m, Each trial type, replotted in mean–variance space. From this picture, it is clear that for this particular set of reward distributions, Fixed odours will be located at the midpoint between Nothing and Variable odours along PC 1, though altering the ratio of mean- to variance-coding neurons will move Fixed odours left or right along PC 1. Different sets of reward distributions could lead to different geometries.n, Mean z-scored firing rates for each neuron, in addition to being higher for rewarded than unrewarded odours (p < 0.001), were also higher for Variable than for Fixed odours (p = 0.006), as assessed by an LME with neuron level observations, averaged over trials, and session-level random effects nested within mouse.o, Same as Extended Data Fig.2o, but for mean. Fraction is higher than chance for both positive- and negative-coding cells (bothp’s < 0.001).Extended Data Fig. 8 REDRL consistently predicts population responses across three additional classical conditioning tasks.一个, Reward distributions for the Bernoulli (顶部), Diverse Distributions (中间), and Fourth Moments (底部）任务。b, Anticipatory lick rate during the late trace period for each task and trial type. (Bernoulli task: 0%,p < 0.001 versus 50, 80, and 100%; 20%,p < 0.001 versus 80 and 100%;50％，p < 0.001 versus 100%; 80%,p = 0.008 versus 100%. Diverse Distributions task: CS 1,

p

 = 0.008 versus CS 2,p < 0.001 versus CS 3–6; CS 2,p < 0.001 versus CS 3–6; CS 3,p = 0.560, 0.243, <0.001 versus CS 4–6, respectively; CS 4,p = 0.560, 0.001 versus CS 5–6, respectively; CS 5,p = 0.009 versus CS 6. Fourth Moments task: Nothing 1 or Nothing 2,p < 0.001 versus Uniform 1, Uniform 2, Bimodal 1, and Bimodal 2; Uniform 1,p = 0.570, 0.336, <0.001 versus Uniform 2, Bimodal 1, and Bimodal 2, respectively; Uniform 2,p = 0.126, <0.001 versus Bimodal 1 and Bimodal 2, respectively; Bimodal 1,p = 0.016 versus Bimodal 2). Dashed line indicates mean reward for that trial, given on the secondaryy-轴。c, 2D PC projections for example sessions in each task.d, 2D PC projections for each model on each of the three tasks.e, Quantification of Pearson correlation between the Euclidean distance matrices measured between each trial type along either PC 1 (左边) or PC 2 (正确的）。(Bernoulli task: PC 1 relative to REDRL,p = 0.994, 0.459, 0.284, <0.001, <0.001, <0.001, 0.861, 0.888, 0.772, <0.001 for Expectile, Quantile, Reflected Quantile, Distributed AU, Partial Distributed AU, AU, Categorical, Laplace, Cumulative, and Moments codes, respectively; PC 2 relative to REDRL,p = 0.666, 0.964, 0.653, <0.001, <0.001, <0.001, <0.001, 0.078, 0.002, <0.001. Diverse Distributions task: PC 1 relative to REDRL,p = 0.999, 0.963, 0.985, <0.001, <0.001, <0.001, <0.001, 0.993, 0.994, 0.011; PC 2 relative to REDRL,p = 0.863, 0.077, 0.050, 0.096, 0.054, 0.147, 0.428, 0.038, 0.065, 0.047. Fourth Moments task: PC 1 relative to REDRL,p = 0.891, 0.990, 0.997, 0.951, 0.928, 0.978, 0.828, 0.984, 0.927, 0.921; PC 2 relative to REDRL,p < 0.001, 0.127, 0.325, 0.167, 0.305, 0.891, 0.839, 0.075, 0.060, 0.021).f, Difference between observed and trial-type shuffled data in the percentage of cells significantly correlating positively or negatively during the late trace period with either mean (左边) or residual variance (正确的）。In the Bernoulli task, mean and variance are orthogonal by design, so residual variance is equivalent to variance.In the Fourth Moments task, mean and variance are fully colinear, so residual variance is always equal to zero.(Bernoulli task:p < 0.001, = 0.013, 0.112, 0.225 for Positive and Negative mean and residual variance differences relative to zero, respectively. Diverse Distributions task:p < 0.001, = 0.009, 0.312, 0.026. Fourth Moments task: both meanp’s < 0.001).g, Pseudo-population parallelism score across subregions in the Fourth Moments task, comparing neural representations of Uniform and Bimodal distributions (relative to chance level of 0:p = 0.291, 0.150, 0.851, 0.002, 0.465, 0.832, 0.775, 0.175, 0.548 for OT, VP, lAcbSh, core, VMS, VLS, DMS, DLS, and All Subregions, respectively. Same order applies to remaining panels in this figure). Pseudo-populations were constructed as in Extended Data Fig.2l, and mAcbSh was excluded because of too few neurons in all animals.h, Same asg, but for CCGP (relative to chance level of 0.5:p = 0.975, 0.997, 0.948, 0.150, 0.852, 0.945, 0.474, 0.693, 0.337).我, Same asg, but for pairwise decoding (Across- vs. Within-distribution:p = 0.893, 0.411, 0.012, 0.184, 0.590, 0.762, 0.256, 0.327, 0.311).j, Same asg, but for congruency analysis (Congruent vs. Incongruent 1:p = 0.457, 0.411, 0.333, 0.606, 0.833, 0.966, 0.956, 0.106, 0.225; Congruent vs. Incongruent 2:p = 0.993, 0.014, 0.265, 0.228, 0.602, 0.978, 0.073, 0.760, 0.007).Extended Data Fig. 9 Additional data for 6-OHDA experiments.一个, Consensus heat map74of all five animals’ lesion locations. 6-OHDA was injected in the lAcbSh but diffused into the VLS, so we considered both regions to be lesioned. We excluded OT, despite the fact that it was often lesioned, because it is not physically contiguous and showed weaker evidence of distributional coding in control animals. The illustration was adapted from ref.74, Elsevier.b, Behavioral decoding analysis comparing fully intact animals (n = 3) and unilaterally lesioned (n = 9) animals across time. For this analysis, animals were considered lesioned if they had received any 6-OHDA injection, even if that hemisphere was never recorded or was mistargeted relative to Neuropixels recording location.c, Quantification of behavioral classifier accuracy during the late trace period. While across-mean behavioral decoding was stronger in the control than the lesioned animals (effect of lesion:p = 0.006, 0.001, 0.173 for Nothing vs. Fixed, Nothing vs. Variable, and Fixed vs. Variable, respectively), both groups of animals clearly learned the task and had above-chance across-mean decoding (p < 0.001 compared to chance level of 50% for both Nothing vs. Fixed and Nothing vs. Variable in control as well as lesioned animals). Interestingly, Fixed vs. Variable classification was also weakly significant (p = 0.032 relative to chance level of 50%) for fully intact control animals, providing behavioral evidence that they did in fact learn this distinction.d, Median fraction deviance explained by the GLM (Extended Data Fig.5) for neurons in control vs. lesioned hemispheres (p = 0.831).

e

, Difference in fraction deviance explained between full model and models in which history (左边^;p^{ = 0.474), reward (}第二;p = 0.623) sensory/reward prediction (第三;p = 0.861) or motor (正确的;p = 0.618) regressors had been dropped out.f, Absolute kernel strength of history (左边;p = 0.634), reward (第二;p = 0.089), expectiles (第三;p = 0.448) or motor (正确的;p = 0.145) regressors.Extended Data Fig. 10 Additional data for two-photon calcium imaging experiments.一个, D1 MSN activity.顶部, heatmaps showing average z-scored deconvolved calcium activity in response to each odour for each neuron, as in Extended Data Fig.2b。底部, grand average z-scored deconvolved calcium activity across all neurons.b, Same as一个, but for D2 MSN activity.c, Anticipatory lick rates for each trial type, computed during the late trace period separately forDrd1-cre和Adora2a-creanimals (in which we imaged D1 or D2 MSNs, respectively).(Drd1-cre, Nothing 1 or Nothing 2:p < 0.001 versus Fixed 1, Fixed 2, Variable 1, and Variable 2;Drd1-cre, Fixed 1:p = 0.960, 0.458, 0.642 versus Fixed 2, Variable 1, and Variable 2, respectively;n = 4 mice, 29 sessions.Adora2a-cre, Nothing 1 or Nothing 2:p < 0.001 versus Fixed 1, Fixed 2, Variable 1, and Variable 2;Adora2a-cre

, Fixed 1:p

 = 0.790, 0.608, 0.686 versus Fixed 2, Variable 1, and Variable 2, respectively;n = 4 mice, 41 sessions. Main effect of genotype, relative to Nothing 1:p = 0.785; interaction of genotype and trial type:p = 0.888, 0.387, 0.525, 0.350, 0.331 for Nothing 2, Fixed 1, Fixed 2, Variable 1, and Variable 2, respectively;n = 8 mice, 70 sessions).如图1C, dashed lines indicate mean reward for that trial type.d, Fraction of neurons whose late trace activity increased (顶部) or decreased (底部) relative to baseline, shown separately for D1 (左边) and D2 (正确的) MSNs and unrewarded (Nothing) versus rewarded (Fixed and Variable) odours (x-轴）;these trial types were pooled before analysis.As expected, a larger fraction of D1 MSNs increases to rewarded rather than unrewarded odours (p = 0.006; mean ± s.e.m. = 0.524 ± 0.074), while there is no difference in the fractions that decrease (p = 0.423; mean ± s.e.m. = –0.098 ± 0.106). Meanwhile, for D2 MSNs, a significantly greater fraction of neurons change their activity on rewarded compared to unrewarded trials, by either increasing (p = 0.022; mean ± s.e.m. = 0.189 ± 0.043) or decreasing (p = 0.016; mean ± s.e.m. = 0.133 ± 0.027) their activity relative to baseline. Asterisks andp-values report the results of paired samples Student’st-tests on rewarded vs. unrewarded fractions across mice.e, REDRL predicts higher variance across trial types for optimistic than for pessimistic reward predictors on average (左边), which is also true in the two-photon data for D1 and D2 MSNs, respectively (正确的）。Small dots are averages within sessions, medium dots are averages within mice, and large dots with error bars show averages ± 95% c.i.across mice (p = 0.017 for effect of genotype).Extended Data Fig. 11 Additional detail for distributional model manipulations.一个, Schematic showing how optogenetic perturbations were simulated for an expectile code (from EDRL). Optimistic (blue) or pessimistic (purple) predictors were shifted from their original values (semi-transparent grey circles) and clamped to low or high values to mimic inhibition (左边, “x”s) or excitation (正确的, triangles), respectively. Panels on the right depict the ground-truth reward distribution, its mean (black line), and the means of the manipulated sets of value predictors (blue or purple dashed lines).b, Same as一个, but for a quantile rather than expectile code.c, Same asb, but for a reflected quantile code. The additional, leftmost panel for each distribution depicts the activity of D1 (yellow) and D2 (green) MSNs at baseline (semi-transparent circles) and after manipulations (opaque “x”s and triangles). These are what are directly clamped by the simulated optogenetic inhibition or excitation. As a result, the effect on the implied value predictors (middle panel) corresponding to D2 MSNs are of opposite sign, as is the change in predicted mean (right panel).d, Same asc, but for the Distributed Actor Uncertainty (AU) model. Since D1 and D2 MSN activities in this model can exceed the maximum reward value, the left panel shows that perturbations were simulated by adding or subtracting a fixed amount from each activity level (opaque “x”s and triangles) relative to baseline (semi-transparent circles). The middle panel plots the resulting value predictors, computed as the pointwise differences between D1 and D2 MSN activities, for pessimistic (purple) and optimistic (blue) manipulations in comparison to baseline (grey semi-transparent circles).e, Same asd, except that only the optimistic or pessimistic half of MSNs were manipulated to simulate perturbations of D1 or D2 MSNs, respectively.f, Same asd, except for the original Actor Uncertainty (AU) model in which there is only one pair of value predictors with balanced learning rates (τ = 0.5).g, Schematic showing how optogenetic perturbations were simulated for a categorical code (from CDRL), which effectively represents the reward distribution using a histogram. Pessimistic (0, 2 μL; purple) or optimistic (6, 8 μL; blue) bins were clamped to 0 or 1 to simulate inhibition or excitation, respectively, relative to baseline (grey). The resulting distributions were normalized to sum to one (see 方法）。Dashed vertical lines show the means of the ground-truth (black) and manipulated distributions.h, Same asg

, except for a Laplace code

40in which each neuron corresponds to the height of 1 – CDF at a particular point. While the baseline case is always monotonically decreasing, simulated excitation or inhibition can change this. Means were computed by differentiating and then normalizing (see 方法）。我, Same ash, except for a cumulative code where each neuron corresponds to the height of the CDF at a particular point.j, Actual differences in lick rate during the last half second of the trace period in response to inhibition of D1 or D2 MSNs, copied from Fig.5f。k, Same asj, but for excitation.l, Predicted difference in mean reward due to inhibition for REDRL and each of the alternative models in一个我。m, Same asl, but for excitation.n, Average lick rates in each group of animals, with (blue and purple) or without (black) manipulations, rarely exceeded 5 Hz.补充信息Supplementary DiscussionThree extensions to the discussion in the main text of the paper, on (1) distinguishing expectile- and quantile-based versions of distributional RL; (2) contrasting our results with non-RPE-based accounts of dopamine; and (3) considering probabilistic coding in the brain more broadly.补充表1Full specification of all linear mixed effects models (LMEs).权利和权限Springer Nature或其许可人（例如，社会或其他合作伙伴）根据与作者或其他权利归属人的出版协议享有本文的独家权利；本文接受的手稿版本的作者自我构造仅受此类出版协议和适用法律的条款的约束。重印和权限关于这篇文章引用本文Lowet, A.S., Zheng, Q., Meng, M.等。^{用于分配加固学习的对手纹状体电路。}自然（2025）。https://doi.org/10.1038/s41586-024-08488-5下载引用已收到：2024年1月2日公认：2024年12月4日出版：2025年2月19日doi：https://doi.org/10.1038/s41586-024-08488-5, Predicted difference in mean reward due to inhibition for REDRL and each of the alternative models in a–i. m, Same as l, but for excitation. n, Average lick rates in each group of animals, with (blue and purple) or without (black) manipulations, rarely exceeded 5 Hz.

Supplementary Discussion

Three extensions to the discussion in the main text of the paper, on (1) distinguishing expectile- and quantile-based versions of distributional RL; (2) contrasting our results with non-RPE-based accounts of dopamine; and (3) considering probabilistic coding in the brain more broadly.

Supplementary Table 1

Full specification of all linear mixed effects models (LMEs).

Cite this article

Lowet, A.S., Zheng, Q., Meng, M. et al. An opponent striatal circuit for distributional reinforcement learning. Nature (2025). https://doi.org/10.1038/s41586-024-08488-5

Download citation

• Received: 02 January 2024

• Accepted: 04 December 2024

• Published: 19 February 2025

• DOI: https://doi.org/10.1038/s41586-024-08488-5