DQN tutorial fixes

[SiftingSands]

Gym's domain name has changed since #1928 (comment)

Minor updates to Gym calls, which would produce errors with the latest version of Gym (0.25.2). (possibly fixes #1432 (comment))

Still doesn't train properly as noted in #1755 (comment) , but at least it runs now as shown in the figure below (didn't touch any other settings in the tutorial).

[facebook-github-bot]

Hi @SiftingSands!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[netlify]

✅ Deploy Preview for pytorch-tutorials-preview ready!

Name	Link
🔨 Latest commit	cf2e13a
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-tutorials-preview/deploys/63176793fd8b890008e7f3f1
😎 Deploy Preview	https://deploy-preview-2026--pytorch-tutorials-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[vmoens]

Thanks @SiftingSands for this
LGTM, if you don't mind I'll have a deeper look at the tuto and check why it doesn't learn before merging.
Forgive me for putting this on hold!

[SiftingSands]

@vmoens No worries at all. I did try tweaking various hyperparameters and doing reward shaping to get it to train within 500 to 1000 episodes. As with others, I saw non-monotonic reward history, where the agent would perform well then forget how to solve the problem again. I believe this is to be expected for vanilla DQN learning from pixels instead of the raw state variables.

I can send over my best results in a new merge request later this evening (GMT -4) if you're interested (still only able to consistently get ~100 timesteps by 1000 episodes, but I haven't tried saving the "best" model to see how it generalizes). I tried to keep my changes minor relative to Adam's original work such as on only using the "duration" in the reward and not using any state variables such as angle or position.

[vmoens]

@vmoens No worries at all. I did try tweaking various hyperparameters and doing reward shaping to get it to train within 500 to 1000 episodes. As with others, I saw non-monotonic reward history, where the agent would perform well then forget how to solve the problem again. I believe this is to be expected for vanilla DQN learning from pixels instead of the raw state variables.

I can send over my best results in a new merge request later this evening (GMT -4) if you're interested (still only able to consistently get ~100 timesteps by 1000 episodes, but I haven't tried saving the "best" model to see how it generalizes). I tried to keep my changes minor relative to Adam's original work such as on only using the "duration" in the reward and not using any state variables such as angle or position.

Sure let's keep iterating on this. There are some things I'm trying ok my side too!
Do you know if it used to train and then it didn't or did we always have this performance?

[SiftingSands]

Sure let's keep iterating on this. There are some things I'm trying ok my side too! Do you know if it used to train and then it didn't or did we always have this performance?

Unfortunately, I do not know if it trained successfully in the past. I looked through a few blogs to see if anyone reported it working, but all successful examples involved significant modifications such as using the state variables instead of pixels. I wonder if it's always been somewhat "broken", since there's this issue from 2018 #209 about missing the target network. Just speculating here though...