The dominant view in neuroscience is that changes in synaptic weights underlie learning. It is unclear, however, how the brain is able to determine which synapses should change, and by how much. This uncertainty stands in sharp contrast to the situation in deep learning, where changes in weights are explicitly engineered to optimize performance. However, the main tool for doing this, backpropagation, does not appear to be biologically plausible, and networks trained with this rule tend to forget old tasks when learning new ones. Here we propose that biological networks instead use a layer-wise learning rule, one motivated by the Gated Linear Network architecture (Veness et al., 2017, 2019). This rule has three features critical for real brains: it allows local learning, even in deep networks; it is resistant to forgetting old tasks when learning new ones; and it is very data efficient – the latter an important requirement for real organisms, which must learn from relatively small training sets. The cost, though, is additional circuitry: dendritic branches must be gated on and off in an input-dependent manner. This gating can be performed by inhibitory neurons, providing them with a novel role in computing. We show that this learning rule performs well – in some cases better than backpropagation – and that it has parallels with the operation of cerebellar circuits.