矩阵求导的一般规律

Consider two vector $x\in {\mathbb{R}}^{d_I}$ and $y\in {\mathbb{R}}^{d_O}$, and $y=Wx$ with $W\in {\mathbb{R}}^{d_o\times d_I}$ .

$$\begin{array}{}\text{(1)}& \begin{array}{rl}& \frac{dy}{dx}\in {\mathbb{R}}^{d^O\times d^I}\\ & \frac{dy}{dx}=W\end{array}\end{array}$$ $$\begin{array}{}\text{(2)}& \begin{array}{rl}& \frac{dy}{dW}\in {\mathbb{R}}^{d^O\times d^O\times d_I}\\ & \frac{dy}{dW}=I_{d_O\times d_O}\otimes x^T\end{array}\end{array}$$

If we define $W=MN$, where $M\in {\mathbb{R}}^{d_O\times r}$ and $N\in {\mathbb{R}}^{r\times d_I}$, we have

$$\begin{array}{}\text{(3)}& \begin{array}{rl}& \frac{dy}{dM}\in {\mathbb{R}}^{d^O\times d^O\times r}\\ & \frac{dy}{dM}=I_{d_O}\otimes (Nx{)}^T=I_{d_O}\otimes (x{)}^T{N}^T\end{array}\end{array}$$

and

$$\begin{array}{}\text{(4)}& \begin{array}{rl}& \frac{dy}{dN}\in {\mathbb{R}}^{d_O\times r\times d_I}\\ & \frac{dy}{dN}=\frac{dy}{dNx}\cdot \frac{dNx}{dN}=M\cdot I_r\otimes x^T\end{array}\end{array}$$

Gradient is different from the derivative. It can be seen as the transpose of derivatives.

$$\begin{array}{}\text{(5)}& {\mathrm{\nabla }}_{x}y=(\frac{\partial y}{\partial x}{)}^T.\end{array}$$

In derivative, we may have:

$$\begin{array}{}\text{(6)}& \frac{dL}{d\theta }=\frac{dL}{df}\cdot \frac{df}{d\theta },\end{array}$$

with the shape $\frac{dL}{d\theta }\in {\mathbb{R}}^{1\times N_{\theta }}$ while in the gradient, it is:

$$\begin{array}{}\text{(7)}& \begin{array}{rl}& {\mathrm{\nabla }}_{\theta }L\in {\mathbb{R}}^{N_{\theta }\times 1}\\ & {\mathrm{\nabla }}_{\theta }L={\mathrm{\nabla }}_{\theta }f\cdot {\mathrm{\nabla }}_{f}L\end{array}\end{array}$$