Missing parameter in math formula

[bingoabs]

Reading Deep Learning with PyTorch: A 60 Minute Blitz, located at line 191-209

$$ J^{T}\cdot \vec{v}=\left(\begin{array}{ccc} \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{1}}\\ \vdots & \ddots & \vdots\\ \frac{\partial y_{1}}{\partial x_{n}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}\right)\left(\begin{array}{c} \frac{\partial l}{\partial y_{1}}\\ \vdots\\ \frac{\partial l}{\partial y_{m}} \end{array}\right)=\left(\begin{array}{c} \frac{\partial l}{\partial x_{1}}\\ \vdots\\ \frac{\partial l}{\partial x_{n}} \end{array}\right) $$

• It looks like we need add a m as multiplier in right side of the fomula, as we have m items to accumulate.

cc @suraj813

[asideofcode-dev]

@svekars @subramen @bingoabs @albanD

The inclusion of the prefactor m in the final term of the vector-Jacobian product is incorrect. The chain rule ensures that the gradient of $$l$$ with respect to $$\mathbf{x}$$, given by $$J^\top \cdot \mathbf{v}$$, naturally accumulates contributions from all intermediate variables $$y_j$$ without requiring any $$m$$-scaling.

The summation over $$j = 1, \dots, m$$ in the vector-Jacobian product aggregates the distinct contributions of each $$y_j$$. Specifically:

$$ \frac{\partial l}{\partial x_i} = \sum_{j=1}^m \frac{\partial l}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_i}, $$

where:

• $$\frac{\partial l}{\partial y_j}$$: Captures how $$l$$ depends on $$y_j$$.
• $$\frac{\partial y_j}{\partial x_i}$$: Captures how $$y_j$$ depends on $$x_i$$.

Each term in the summation is distinct, reflecting the unique paths through which $$x_i$$ influences $$l$$ via $$y_j$$. Importantly, $$\frac{\partial l}{\partial y_j} \cdot \frac{\partial y_j}{\partial x_i}$$ does not directly reduce to $$\frac{\partial l}{\partial x_i}$$, as it captures only one pathway of dependence. The chain rule ensures that all such pathways are summed to yield the correct gradient.

Matrix-vector multiplication $$J^\top \cdot \mathbf{v}$$ performs this aggregation automatically, without redundancy or repetition that would require explicit scaling by $$m$$. Introducing $$m$$ incorrectly suggests that all terms are identical, which is not the case.

Thus, the correct expression for the gradient is:

$$ J^\top \cdot \mathbf{v} = \frac{\partial l}{\partial \mathbf{x}} , $$

where $$\mathbf{v} = \frac{\partial l}{\partial \mathbf{y}}$$, without any additional $$m$$-prefactor.

[albanD]

Ho yeah, I definitely accepted this PR too fast. Thanks for catching this and sorry for the confusion!