You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
"\nAutograd: automatic differentiation\n===================================\n\nCentral to all neural networks in PyTorch is the ``autograd`` package.\nLet\u2019s first briefly visit this, and we will then go to training our\nfirst neural network.\n\n\nThe ``autograd`` package provides automatic differentiation for all operations\non Tensors. It is a define-by-run framework, which means that your backprop is\ndefined by how your code is run, and that every single iteration can be\ndifferent.\n\nLet us see this in more simple terms with some examples.\n\nVariable\n--------\n\n``autograd.Variable`` is the central class of the package. It wraps a\nTensor, and supports nearly all of operations defined on it. Once you\nfinish your computation you can call ``.backward()`` and have all the\ngradients computed automatically.\n\nYou can access the raw tensor through the ``.data`` attribute, while the\ngradient w.r.t. this variable is accumulated into ``.grad``.\n\n.. figure:: /_static/img/Variable.png\n :alt: Variable\n\n Variable\n\nThere\u2019s one more class which is very important for autograd\nimplementation - a ``Function``.\n\n``Variable`` and ``Function`` are interconnected and build up an acyclic\ngraph, that encodes a complete history of computation. Each variable has\na ``.creator`` attribute that references a ``Function`` that has created\nthe ``Variable`` (except for Variables created by the user - their\n``creator is None``).\n\nIf you want to compute the derivatives, you can call ``.backward()`` on\na ``Variable``. If ``Variable`` is a scalar (i.e. it holds a one element\ndata), you don\u2019t need to specify any arguments to ``backward()``,\nhowever if it has more elements, you need to specify a ``grad_output``\nargument that is a tensor of matching shape.\n\n"
"``y`` was created as a result of an operation, so it has a creator.\n\n"
94
73
]
95
74
},
96
75
{
76
+
"cell_type": "code",
77
+
"outputs": [],
97
78
"metadata": {
98
79
"collapsed": false
99
80
},
100
-
"outputs": [],
101
-
"cell_type": "code",
102
-
"execution_count": null,
103
81
"source": [
104
82
"print(y.creator)"
105
-
]
83
+
],
84
+
"execution_count": null
106
85
},
107
86
{
108
-
"metadata": {},
109
87
"cell_type": "markdown",
88
+
"metadata": {},
110
89
"source": [
111
90
"Do more operations on y\n\n"
112
91
]
113
92
},
114
93
{
94
+
"cell_type": "code",
95
+
"outputs": [],
115
96
"metadata": {
116
97
"collapsed": false
117
98
},
118
-
"outputs": [],
119
-
"cell_type": "code",
120
-
"execution_count": null,
121
99
"source": [
122
100
"z = y * y * 3\nout = z.mean()\n\nprint(z, out)"
123
-
]
101
+
],
102
+
"execution_count": null
124
103
},
125
104
{
126
-
"metadata": {},
127
105
"cell_type": "markdown",
106
+
"metadata": {},
128
107
"source": [
129
108
"Gradients\n---------\nlet's backprop now\n``out.backward()`` is equivalent to doing ``out.backward(torch.Tensor([1.0]))``\n\n"
130
109
]
131
110
},
132
111
{
112
+
"cell_type": "code",
113
+
"outputs": [],
133
114
"metadata": {
134
115
"collapsed": false
135
116
},
136
-
"outputs": [],
137
-
"cell_type": "code",
138
-
"execution_count": null,
139
117
"source": [
140
118
"out.backward()"
141
-
]
119
+
],
120
+
"execution_count": null
142
121
},
143
122
{
144
-
"metadata": {},
145
123
"cell_type": "markdown",
124
+
"metadata": {},
146
125
"source": [
147
126
"print gradients d(out)/dx\n\n\n"
148
127
]
149
128
},
150
129
{
130
+
"cell_type": "code",
131
+
"outputs": [],
151
132
"metadata": {
152
133
"collapsed": false
153
134
},
154
-
"outputs": [],
155
-
"cell_type": "code",
156
-
"execution_count": null,
157
135
"source": [
158
136
"print(x.grad)"
159
-
]
137
+
],
138
+
"execution_count": null
160
139
},
161
140
{
162
-
"metadata": {},
163
141
"cell_type": "markdown",
142
+
"metadata": {},
164
143
"source": [
165
144
"You should have got a matrix of ``4.5``. Let\u2019s call the ``out``\n*Variable* \u201c$o$\u201d.\nWe have that $o = \\frac{1}{4}\\sum_i z_i$,\n $z_i = 3(x_i+2)^2$ and $z_i\\bigr\\rvert_{x_i=1} = 27$.\n Therefore,\n $\\frac{\\partial o}{\\partial x_i} = \\frac{3}{2}(x_i+2)$, hence\n $\\frac{\\partial o}{\\partial x_i}\\bigr\\rvert_{x_i=1} = \\frac{9}{2} = 4.5$.\n\n"
166
145
]
167
146
},
168
147
{
169
-
"metadata": {},
170
148
"cell_type": "markdown",
149
+
"metadata": {},
171
150
"source": [
172
151
"You can do many crazy things with autograd!\n\n"
173
152
]
174
153
},
175
154
{
155
+
"cell_type": "code",
156
+
"outputs": [],
176
157
"metadata": {
177
158
"collapsed": false
178
159
},
179
-
"outputs": [],
180
-
"cell_type": "code",
181
-
"execution_count": null,
182
160
"source": [
183
161
"x = torch.randn(3)\nx = Variable(x, requires_grad=True)\n\ny = x * 2\nwhile y.data.norm() < 1000:\n y = y * 2\n\nprint(y)"
0 commit comments