profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/muupan/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

muupan/deep-reinforcement-learning-papers 801

A list of papers and resources dedicated to deep reinforcement learning

muupan/async-rl 388

Replicating "Asynchronous Methods for Deep Reinforcement Learning" (http://arxiv.org/abs/1602.01783)

muupan/dqn-in-the-caffe 213

An implementation of Deep Q-Network using Caffe

muupan/deep-ensemble-uncertainty 29

An implementation of "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles" (http://arxiv.org/abs/1612.01474)

muupan/caffe 9

Caffe: a fast framework for deep learning. For the most recent version checkout the dev branch. For the latest stable release checkout the master branch.

muupan/chainer-cocob 6

COCOB-Backprop (https://arxiv.org/abs/1705.07795) implementation for Chainer

muupan/chainerrl 2

ChainerRL is a deep reinforcement learning library built on top of Chainer.

muupan/ggpe 2

General Game Playing Engine in C++11 using YAP Prolog

muupan/chainer-elu 1

Chainer implementation of Exponential Linear Unit (ELU)

muupan/chainer-entropy-adam 1

Chainer-based implementation of Entropy-Adam https://arxiv.org/abs/1611.01838

startednttcslab/japanese-dialog-transformers

started time in 7 hours

startedgoogle-research/seed_rl

started time in 2 days

pull request commentpfnet/pfrl

Snapshot for preemption

/test

knshnb

comment created time in 6 days

startedj3soon/arxiv-utils

started time in 6 days

startedTylerGubala/blenderpy

started time in 7 days

startedssloy/tinyrenderer

started time in 9 days

startedblender/blender

started time in 11 days

delete branch muupan/pfrl

delete branch : fix-ci-monitor-test

delete time in 13 days

pull request commentpfnet/pfrl

[WIP] Fix CI error due to gym's version

/test

muupan

comment created time in 13 days

PR opened pfnet/pfrl

[WIP] Fix CI error due to gym's version

Resolves #158

Both atari_py and opencv-python are included in gym[atari]. See https://github.com/openai/gym/blob/a4c3f15e9b0c7525ec9006e92a3bf6bc8bd0a798/setup.py#L11-L19

+1 -1

0 comment

1 changed file

pr created time in 13 days

create barnchmuupan/pfrl

branch : fix-ci-monitor-test

created branch time in 13 days

issue closedpfnet/pfrl

[BUG] lazy execution issue with lambda

https://github.com/pfnet/pfrl/blob/7b0c7e938ba2c0c56a941c766c68635d0dad43c8/examples/atari/train_ppo_ale.py#L199-L200

On the line above, there is a lazy execution issue with lambda

when called like this with list comprehension, the expression (lambda: make_env(idx, test)) is never actually evaluated until it's been called. When it is being called in pfrl.envs.MultiprocessVectorEnv, the for loop has looped through and idx == args.num_envs for all the lambdas in the list

As a result, all the envs in the MultiprocessVectorEnv are actually seeded by the same seed.

closed time in 13 days

zhouzypaul

issue commentpfnet/pfrl

[BUG] lazy execution issue with lambda

resolved by https://github.com/pfnet/pfrl/pull/157

zhouzypaul

comment created time in 13 days

push eventpfnet/pfrl

Paul Zhou

commit sha f8a561ae627d31138ac6fe975e06bfbfe20ee345

[BUGFIX] use functools.partial instead of lambda, so there's no lazy execution issue when assigning env seeds

view details

Yasuhiro Fujita

commit sha 87824850a91589e43659a0802b009a41bd7c84e9

Merge pull request #157 from zhouzypaul/master [BUGFIX] use functools.partial instead of lambda

view details

push time in 13 days

PR merged pfnet/pfrl

[BUGFIX] use functools.partial instead of lambda

so there's no lazy execution issue when assigning env seeds

+2 -1

5 comments

1 changed file

zhouzypaul

pr closed time in 13 days

pull request commentpfnet/pfrl

[BUGFIX] use functools.partial instead of lambda

pfrl/tests/wrappers_tests/test_monitor.py failed in pfn-public-ci/pfrl.cpu, but this must be unrelated (https://github.com/pfnet/pfrl/issues/158) to the PR so I'll merge.

zhouzypaul

comment created time in 13 days

issue openedpfnet/pfrl

`pfrl/tests/wrappers_tests/test_monitor.py` fails in pfn-public-ci/pfrl.cpu

I haven't checked it yet but it seems like the issue is related to gym, perhaps the new version of it.

00:06:15.994491 STDOUT 2254] | [FAILED] /pfrl/tests/wrappers_tests/test_monitor.py (4 failed in 2.70 seconds * 2 trials) |  
-- | -- | --
00:06:15.994594 STDOUT 2254] | ============================= test session starts ============================== |  
00:06:15.994597 STDOUT 2254] | platform linux -- Python 3.7.7, pytest-4.1.1, py-1.10.0, pluggy-1.0.0 |  
00:06:15.994602 STDOUT 2254] | rootdir: /pfrl, inifile: pytest.ini |  
00:06:15.994747 STDOUT 2254] | plugins: xdist-1.26.1, forked-1.3.0 |  
00:06:15.994753 STDOUT 2254] | collected 4 items |  
00:06:15.994765 STDOUT 2254] |   |  
00:06:15.994832 STDOUT 2254] | pfrl/tests/wrappers_tests/test_monitor.py FFFF                           [100%] |  
00:06:15.994835 STDOUT 2254] |   |  
00:06:15.994839 STDOUT 2254] | =================================== FAILURES =================================== |  
00:06:15.994866 STDOUT 2254] | _______________________________ test_monitor[1] ________________________________ |  
00:06:15.994868 STDOUT 2254] |   |  
00:06:15.994871 STDOUT 2254] | """ |  
00:06:15.994901 STDOUT 2254] | import os |  
00:06:15.994904 STDOUT 2254] | import sys |  
00:06:15.994910 STDOUT 2254] |   |  
00:06:15.994941 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:15.994943 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:15.994948 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:15.994993 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:15.994996 STDOUT 2254] |   |  
00:06:15.995001 STDOUT 2254] | from gym import error |  
00:06:15.995070 STDOUT 2254] |   |  
00:06:15.995073 STDOUT 2254] | try: |  
00:06:15.995108 STDOUT 2254] | >       import pyglet |  
00:06:15.995112 STDOUT 2254] | E       ModuleNotFoundError: No module named 'pyglet' |  
00:06:15.995120 STDOUT 2254] |   |  
00:06:15.995167 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:15: ModuleNotFoundError |  
00:06:15.995170 STDOUT 2254] |   |  
00:06:15.995173 STDOUT 2254] | During handling of the above exception, another exception occurred: |  
00:06:15.995194 STDOUT 2254] |   |  
00:06:15.995195 STDOUT 2254] | n_episodes = 1 |  
00:06:15.995199 STDOUT 2254] |   |  
00:06:15.995211 STDOUT 2254] | @pytest.mark.parametrize("n_episodes", [1, 2, 3, 4]) |  
00:06:15.995212 STDOUT 2254] | def test_monitor(n_episodes): |  
00:06:15.995215 STDOUT 2254] | steps = 15 |  
00:06:15.995243 STDOUT 2254] |   |  
00:06:15.995246 STDOUT 2254] | env = gym.make("CartPole-v1") |  
00:06:15.995251 STDOUT 2254] | # unwrap default TimeLimit and wrap with new one to simulate done=True |  
00:06:15.995303 STDOUT 2254] | # at step 5 |  
00:06:15.995307 STDOUT 2254] | assert isinstance(env, TimeLimit) |  
00:06:15.995313 STDOUT 2254] | env = env.env  # unwrap |  
00:06:15.995377 STDOUT 2254] | env = TimeLimit(env, max_episode_steps=5)  # wrap |  
00:06:15.995381 STDOUT 2254] |   |  
00:06:15.995386 STDOUT 2254] | tmpdir = tempfile.mkdtemp() |  
00:06:15.995425 STDOUT 2254] | try: |  
00:06:15.995428 STDOUT 2254] | env = pfrl.wrappers.Monitor( |  
00:06:15.995433 STDOUT 2254] | env, directory=tmpdir, video_callable=lambda episode_id: True |  
00:06:15.995512 STDOUT 2254] | ) |  
00:06:15.995514 STDOUT 2254] | episode_idx = 0 |  
00:06:16.330165 STDOUT 2254] | episode_len = 0 |  
00:06:16.330255 STDOUT 2254] | t = 0 |  
00:06:16.330259 STDOUT 2254] | >           _ = env.reset() |  
00:06:16.330265 STDOUT 2254] |   |  
00:06:16.330351 STDOUT 2254] | pfrl/tests/wrappers_tests/test_monitor.py:31: |  
00:06:16.330354 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:16.330358 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:54: in reset |  
00:06:16.330390 STDOUT 2254] | self._after_reset(observation) |  
00:06:16.330393 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:239: in _after_reset |  
00:06:16.330400 STDOUT 2254] | self.reset_video_recorder() |  
00:06:16.330507 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:265: in reset_video_recorder |  
00:06:16.330511 STDOUT 2254] | self.video_recorder.capture_frame() |  
00:06:16.330517 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/video_recorder.py:125: in capture_frame |  
00:06:16.330570 STDOUT 2254] | frame = self.env.render(mode=render_mode) |  
00:06:16.330573 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/core.py:254: in render |  
00:06:16.330580 STDOUT 2254] | return self.env.render(mode, **kwargs) |  
00:06:16.330611 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/cartpole.py:179: in render |  
00:06:16.330616 STDOUT 2254] | from gym.envs.classic_control import rendering |  
00:06:16.330622 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:16.330668 STDOUT 2254] |   |  
00:06:16.330671 STDOUT 2254] | """ |  
00:06:16.330676 STDOUT 2254] | import os |  
00:06:16.330707 STDOUT 2254] | import sys |  
00:06:16.330710 STDOUT 2254] |   |  
00:06:16.330715 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:16.330784 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:16.330787 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:16.330793 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:16.330824 STDOUT 2254] |   |  
00:06:16.330826 STDOUT 2254] | from gym import error |  
00:06:16.330830 STDOUT 2254] |   |  
00:06:16.330852 STDOUT 2254] | try: |  
00:06:16.330854 STDOUT 2254] | import pyglet |  
00:06:16.330859 STDOUT 2254] | except ImportError as e: |  
00:06:16.330876 STDOUT 2254] | raise ImportError( |  
00:06:16.330879 STDOUT 2254] | """ |  
00:06:16.330885 STDOUT 2254] | Cannot import pyglet. |  
00:06:16.330925 STDOUT 2254] | HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:16.330927 STDOUT 2254] | But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:16.330932 STDOUT 2254] | 'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:16.330977 STDOUT 2254] | >       """ |  
00:06:16.667513 STDOUT 2254] | ) |  
00:06:16.667634 STDOUT 2254] | E       ImportError: |  
00:06:16.667637 STDOUT 2254] | E           Cannot import pyglet. |  
00:06:16.667643 STDOUT 2254] | E           HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:16.667691 STDOUT 2254] | E           But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:16.667696 STDOUT 2254] | E           'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:16.667703 STDOUT 2254] |   |  
00:06:16.667760 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:23: ImportError |  
00:06:16.667762 STDOUT 2254] | _______________________________ test_monitor[2] ________________________________ |  
00:06:16.667766 STDOUT 2254] |   |  
00:06:16.667785 STDOUT 2254] | """ |  
00:06:16.667787 STDOUT 2254] | import os |  
00:06:16.667790 STDOUT 2254] | import sys |  
00:06:16.667815 STDOUT 2254] |   |  
00:06:16.667817 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:16.667821 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:16.667843 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:16.667846 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:16.667849 STDOUT 2254] |   |  
00:06:16.667876 STDOUT 2254] | from gym import error |  
00:06:16.667877 STDOUT 2254] |   |  
00:06:16.667881 STDOUT 2254] | try: |  
00:06:16.667921 STDOUT 2254] | >       import pyglet |  
00:06:16.667925 STDOUT 2254] | E       ModuleNotFoundError: No module named 'pyglet' |  
00:06:16.667932 STDOUT 2254] |   |  
00:06:16.667974 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:15: ModuleNotFoundError |  
00:06:16.667977 STDOUT 2254] |   |  
00:06:16.667982 STDOUT 2254] | During handling of the above exception, another exception occurred: |  
00:06:16.668028 STDOUT 2254] |   |  
00:06:16.668030 STDOUT 2254] | n_episodes = 2 |  
00:06:16.668035 STDOUT 2254] |   |  
00:06:16.668058 STDOUT 2254] | @pytest.mark.parametrize("n_episodes", [1, 2, 3, 4]) |  
00:06:16.668060 STDOUT 2254] | def test_monitor(n_episodes): |  
00:06:16.668063 STDOUT 2254] | steps = 15 |  
00:06:16.668093 STDOUT 2254] |   |  
00:06:16.668094 STDOUT 2254] | env = gym.make("CartPole-v1") |  
00:06:16.668098 STDOUT 2254] | # unwrap default TimeLimit and wrap with new one to simulate done=True |  
00:06:16.668122 STDOUT 2254] | # at step 5 |  
00:06:16.668123 STDOUT 2254] | assert isinstance(env, TimeLimit) |  
00:06:16.668126 STDOUT 2254] | env = env.env  # unwrap |  
00:06:16.668149 STDOUT 2254] | env = TimeLimit(env, max_episode_steps=5)  # wrap |  
00:06:16.668150 STDOUT 2254] |   |  
00:06:16.668153 STDOUT 2254] | tmpdir = tempfile.mkdtemp() |  
00:06:16.668225 STDOUT 2254] | try: |  
00:06:16.668229 STDOUT 2254] | env = pfrl.wrappers.Monitor( |  
00:06:16.668234 STDOUT 2254] | env, directory=tmpdir, video_callable=lambda episode_id: True |  
00:06:16.668298 STDOUT 2254] | ) |  
00:06:17.001995 STDOUT 2254] | episode_idx = 0 |  
00:06:17.001999 STDOUT 2254] | episode_len = 0 |  
00:06:17.002005 STDOUT 2254] | t = 0 |  
00:06:17.002064 STDOUT 2254] | >           _ = env.reset() |  
00:06:17.002066 STDOUT 2254] |   |  
00:06:17.002071 STDOUT 2254] | pfrl/tests/wrappers_tests/test_monitor.py:31: |  
00:06:17.002099 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:17.002101 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:54: in reset |  
00:06:17.002107 STDOUT 2254] | self._after_reset(observation) |  
00:06:17.002153 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:239: in _after_reset |  
00:06:17.002156 STDOUT 2254] | self.reset_video_recorder() |  
00:06:17.002161 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:265: in reset_video_recorder |  
00:06:17.002202 STDOUT 2254] | self.video_recorder.capture_frame() |  
00:06:17.002204 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/video_recorder.py:125: in capture_frame |  
00:06:17.002210 STDOUT 2254] | frame = self.env.render(mode=render_mode) |  
00:06:17.002272 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/core.py:254: in render |  
00:06:17.002277 STDOUT 2254] | return self.env.render(mode, **kwargs) |  
00:06:17.002283 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/cartpole.py:179: in render |  
00:06:17.002338 STDOUT 2254] | from gym.envs.classic_control import rendering |  
00:06:17.002341 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:17.002356 STDOUT 2254] |   |  
00:06:17.002400 STDOUT 2254] | """ |  
00:06:17.002403 STDOUT 2254] | import os |  
00:06:17.002409 STDOUT 2254] | import sys |  
00:06:17.002445 STDOUT 2254] |   |  
00:06:17.002448 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:17.002468 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:17.002470 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:17.002474 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:17.002501 STDOUT 2254] |   |  
00:06:17.002503 STDOUT 2254] | from gym import error |  
00:06:17.002507 STDOUT 2254] |   |  
00:06:17.002530 STDOUT 2254] | try: |  
00:06:17.002531 STDOUT 2254] | import pyglet |  
00:06:17.002544 STDOUT 2254] | except ImportError as e: |  
00:06:17.002588 STDOUT 2254] | raise ImportError( |  
00:06:17.002590 STDOUT 2254] | """ |  
00:06:17.002592 STDOUT 2254] | Cannot import pyglet. |  
00:06:17.002625 STDOUT 2254] | HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:17.002628 STDOUT 2254] | But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:17.002709 STDOUT 2254] | 'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:17.002711 STDOUT 2254] | >       """ |  
00:06:17.336380 STDOUT 2254] | ) |  
00:06:17.336434 STDOUT 2254] | E       ImportError: |  
00:06:17.336438 STDOUT 2254] | E           Cannot import pyglet. |  
00:06:17.336443 STDOUT 2254] | E           HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:17.336500 STDOUT 2254] | E           But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:17.336503 STDOUT 2254] | E           'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:17.336508 STDOUT 2254] |   |  
00:06:17.336532 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:23: ImportError |  
00:06:17.336538 STDOUT 2254] | _______________________________ test_monitor[3] ________________________________ |  
00:06:17.336547 STDOUT 2254] |   |  
00:06:17.336572 STDOUT 2254] | """ |  
00:06:17.336574 STDOUT 2254] | import os |  
00:06:17.336578 STDOUT 2254] | import sys |  
00:06:17.336628 STDOUT 2254] |   |  
00:06:17.336631 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:17.336635 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:17.336687 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:17.336690 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:17.336695 STDOUT 2254] |   |  
00:06:17.336815 STDOUT 2254] | from gym import error |  
00:06:17.336819 STDOUT 2254] |   |  
00:06:17.336825 STDOUT 2254] | try: |  
00:06:17.336879 STDOUT 2254] | >       import pyglet |  
00:06:17.336881 STDOUT 2254] | E       ModuleNotFoundError: No module named 'pyglet' |  
00:06:17.336885 STDOUT 2254] |   |  
00:06:17.336907 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:15: ModuleNotFoundError |  
00:06:17.336910 STDOUT 2254] |   |  
00:06:17.336913 STDOUT 2254] | During handling of the above exception, another exception occurred: |  
00:06:17.336931 STDOUT 2254] |   |  
00:06:17.336933 STDOUT 2254] | n_episodes = 3 |  
00:06:17.336936 STDOUT 2254] |   |  
00:06:17.336960 STDOUT 2254] | @pytest.mark.parametrize("n_episodes", [1, 2, 3, 4]) |  
00:06:17.336963 STDOUT 2254] | def test_monitor(n_episodes): |  
00:06:17.336968 STDOUT 2254] | steps = 15 |  
00:06:17.337014 STDOUT 2254] |   |  
00:06:17.337017 STDOUT 2254] | env = gym.make("CartPole-v1") |  
00:06:17.337022 STDOUT 2254] | # unwrap default TimeLimit and wrap with new one to simulate done=True |  
00:06:17.337071 STDOUT 2254] | # at step 5 |  
00:06:17.337073 STDOUT 2254] | assert isinstance(env, TimeLimit) |  
00:06:17.337080 STDOUT 2254] | env = env.env  # unwrap |  
00:06:17.337119 STDOUT 2254] | env = TimeLimit(env, max_episode_steps=5)  # wrap |  
00:06:17.337122 STDOUT 2254] |   |  
00:06:17.337128 STDOUT 2254] | tmpdir = tempfile.mkdtemp() |  
00:06:17.337165 STDOUT 2254] | try: |  
00:06:17.337167 STDOUT 2254] | env = pfrl.wrappers.Monitor( |  
00:06:17.337172 STDOUT 2254] | env, directory=tmpdir, video_callable=lambda episode_id: True |  
00:06:17.670338 STDOUT 2254] | ) |  
00:06:17.670342 STDOUT 2254] | episode_idx = 0 |  
00:06:17.670348 STDOUT 2254] | episode_len = 0 |  
00:06:17.670404 STDOUT 2254] | t = 0 |  
00:06:17.670409 STDOUT 2254] | >           _ = env.reset() |  
00:06:17.670414 STDOUT 2254] |   |  
00:06:17.670472 STDOUT 2254] | pfrl/tests/wrappers_tests/test_monitor.py:31: |  
00:06:17.670474 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:17.670480 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:54: in reset |  
00:06:17.670572 STDOUT 2254] | self._after_reset(observation) |  
00:06:17.670575 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:239: in _after_reset |  
00:06:17.670581 STDOUT 2254] | self.reset_video_recorder() |  
00:06:17.670682 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:265: in reset_video_recorder |  
00:06:17.670686 STDOUT 2254] | self.video_recorder.capture_frame() |  
00:06:17.670691 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/video_recorder.py:125: in capture_frame |  
00:06:17.670731 STDOUT 2254] | frame = self.env.render(mode=render_mode) |  
00:06:17.670748 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/core.py:254: in render |  
00:06:17.670754 STDOUT 2254] | return self.env.render(mode, **kwargs) |  
00:06:17.670790 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/cartpole.py:179: in render |  
00:06:17.670793 STDOUT 2254] | from gym.envs.classic_control import rendering |  
00:06:17.670797 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:17.670831 STDOUT 2254] |   |  
00:06:17.670834 STDOUT 2254] | """ |  
00:06:17.670838 STDOUT 2254] | import os |  
00:06:17.670867 STDOUT 2254] | import sys |  
00:06:17.670870 STDOUT 2254] |   |  
00:06:17.670875 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:17.670906 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:17.670909 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:17.670913 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:17.670948 STDOUT 2254] |   |  
00:06:17.670951 STDOUT 2254] | from gym import error |  
00:06:17.670957 STDOUT 2254] |   |  
00:06:17.670978 STDOUT 2254] | try: |  
00:06:17.671008 STDOUT 2254] | import pyglet |  
00:06:17.671011 STDOUT 2254] | except ImportError as e: |  
00:06:17.671017 STDOUT 2254] | raise ImportError( |  
00:06:17.671047 STDOUT 2254] | """ |  
00:06:17.671050 STDOUT 2254] | Cannot import pyglet. |  
00:06:17.671056 STDOUT 2254] | HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:17.671127 STDOUT 2254] | But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:17.671130 STDOUT 2254] | 'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:18.003967 STDOUT 2254] | >       """ |  
00:06:18.004058 STDOUT 2254] | ) |  
00:06:18.004065 STDOUT 2254] | E       ImportError: |  
00:06:18.004071 STDOUT 2254] | E           Cannot import pyglet. |  
00:06:18.004112 STDOUT 2254] | E           HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:18.004115 STDOUT 2254] | E           But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:18.004121 STDOUT 2254] | E           'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:18.004180 STDOUT 2254] |   |  
00:06:18.004183 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:23: ImportError |  
00:06:18.004251 STDOUT 2254] | _______________________________ test_monitor[4] ________________________________ |  
00:06:18.004255 STDOUT 2254] |   |  
00:06:18.004261 STDOUT 2254] | """ |  
00:06:18.004289 STDOUT 2254] | import os |  
00:06:18.004292 STDOUT 2254] | import sys |  
00:06:18.004297 STDOUT 2254] |   |  
00:06:18.004333 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:18.004336 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:18.004342 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:18.004394 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:18.004399 STDOUT 2254] |   |  
00:06:18.004404 STDOUT 2254] | from gym import error |  
00:06:18.004453 STDOUT 2254] |   |  
00:06:18.004455 STDOUT 2254] | try: |  
00:06:18.004459 STDOUT 2254] | >       import pyglet |  
00:06:18.004492 STDOUT 2254] | E       ModuleNotFoundError: No module named 'pyglet' |  
00:06:18.004494 STDOUT 2254] |   |  
00:06:18.004521 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:15: ModuleNotFoundError |  
00:06:18.004524 STDOUT 2254] |   |  
00:06:18.004529 STDOUT 2254] | During handling of the above exception, another exception occurred: |  
00:06:18.004561 STDOUT 2254] |   |  
00:06:18.004564 STDOUT 2254] | n_episodes = 4 |  
00:06:18.004568 STDOUT 2254] |   |  
00:06:18.004574 STDOUT 2254] | @pytest.mark.parametrize("n_episodes", [1, 2, 3, 4]) |  
00:06:18.004592 STDOUT 2254] | def test_monitor(n_episodes): |  
00:06:18.004595 STDOUT 2254] | steps = 15 |  
00:06:18.004601 STDOUT 2254] |   |  
00:06:18.004635 STDOUT 2254] | env = gym.make("CartPole-v1") |  
00:06:18.004637 STDOUT 2254] | # unwrap default TimeLimit and wrap with new one to simulate done=True |  
00:06:18.004640 STDOUT 2254] | # at step 5 |  
00:06:18.004699 STDOUT 2254] | assert isinstance(env, TimeLimit) |  
00:06:18.004703 STDOUT 2254] | env = env.env  # unwrap |  
00:06:18.004709 STDOUT 2254] | env = TimeLimit(env, max_episode_steps=5)  # wrap |  
00:06:18.004746 STDOUT 2254] |   |  
00:06:18.004749 STDOUT 2254] | tmpdir = tempfile.mkdtemp() |  
00:06:18.004753 STDOUT 2254] | try: |  
00:06:18.004848 STDOUT 2254] | env = pfrl.wrappers.Monitor( |  
00:06:18.344241 STDOUT 2254] | env, directory=tmpdir, video_callable=lambda episode_id: True |  
00:06:18.344247 STDOUT 2254] | ) |  
00:06:18.344253 STDOUT 2254] | episode_idx = 0 |  
00:06:18.344342 STDOUT 2254] | episode_len = 0 |  
00:06:18.344345 STDOUT 2254] | t = 0 |  
00:06:18.344350 STDOUT 2254] | >           _ = env.reset() |  
00:06:18.344453 STDOUT 2254] |   |  
00:06:18.344455 STDOUT 2254] | pfrl/tests/wrappers_tests/test_monitor.py:31: |  
00:06:18.344459 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:18.344480 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:54: in reset |  
00:06:18.344483 STDOUT 2254] | self._after_reset(observation) |  
00:06:18.344486 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:239: in _after_reset |  
00:06:18.344515 STDOUT 2254] | self.reset_video_recorder() |  
00:06:18.344517 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py:265: in reset_video_recorder |  
00:06:18.344520 STDOUT 2254] | self.video_recorder.capture_frame() |  
00:06:18.344543 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/video_recorder.py:125: in capture_frame |  
00:06:18.344544 STDOUT 2254] | frame = self.env.render(mode=render_mode) |  
00:06:18.344547 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/core.py:254: in render |  
00:06:18.344568 STDOUT 2254] | return self.env.render(mode, **kwargs) |  
00:06:18.344570 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/cartpole.py:179: in render |  
00:06:18.344572 STDOUT 2254] | from gym.envs.classic_control import rendering |  
00:06:18.344594 STDOUT 2254] | _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ |  
00:06:18.344596 STDOUT 2254] |   |  
00:06:18.344598 STDOUT 2254] | """ |  
00:06:18.344651 STDOUT 2254] | import os |  
00:06:18.344653 STDOUT 2254] | import sys |  
00:06:18.344657 STDOUT 2254] |   |  
00:06:18.344688 STDOUT 2254] | if "Apple" in sys.version: |  
00:06:18.344690 STDOUT 2254] | if "DYLD_FALLBACK_LIBRARY_PATH" in os.environ: |  
00:06:18.344693 STDOUT 2254] | os.environ["DYLD_FALLBACK_LIBRARY_PATH"] += ":/usr/lib" |  
00:06:18.344719 STDOUT 2254] | # (JDS 2016/04/15): avoid bug on Anaconda 2.3.0 / Yosemite |  
00:06:18.344721 STDOUT 2254] |   |  
00:06:18.344725 STDOUT 2254] | from gym import error |  
00:06:18.344748 STDOUT 2254] |   |  
00:06:18.344750 STDOUT 2254] | try: |  
00:06:18.344753 STDOUT 2254] | import pyglet |  
00:06:18.344790 STDOUT 2254] | except ImportError as e: |  
00:06:18.344791 STDOUT 2254] | raise ImportError( |  
00:06:18.344797 STDOUT 2254] | """ |  
00:06:18.344819 STDOUT 2254] | Cannot import pyglet. |  
00:06:18.344820 STDOUT 2254] | HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:18.344823 STDOUT 2254] | But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:18.344868 STDOUT 2254] | 'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:18.678537 STDOUT 2254] | >       """ |  
00:06:18.678604 STDOUT 2254] | ) |  
00:06:18.678606 STDOUT 2254] | E       ImportError: |  
00:06:18.678610 STDOUT 2254] | E           Cannot import pyglet. |  
00:06:18.678635 STDOUT 2254] | E           HINT: you can install pyglet directly via 'pip install pyglet'. |  
00:06:18.678636 STDOUT 2254] | E           But if you really just want to install all Gym dependencies and not have to think about it, |  
00:06:18.678640 STDOUT 2254] | E           'pip install -e .[all]' or 'pip install gym[all]' will do it. |  
00:06:18.678665 STDOUT 2254] |   |  
00:06:18.678667 STDOUT 2254] | opt/conda/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py:23: ImportError |  
00:06:18.678670 STDOUT 2254] | =========================== 4 failed in 2.70 seconds =========================== |  
00:06:18.678692 STDOUT 2254] |   |  
00:06:18.678693 STDOUT 2254] | Exception ignored in: <function Monitor.__del__ at 0x7f068a825a70> |  
00:06:18.678696 STDOUT 2254] | Traceback (most recent call last): |  
00:06:18.678716 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 287, in __del__ |  
00:06:18.678718 STDOUT 2254] | self.close() |  
00:06:18.678721 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 180, in close |  
00:06:18.678838 STDOUT 2254] | self.stats_recorder.close() |  
00:06:18.678843 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/pfrl/wrappers/monitor.py", line 99, in close |  
00:06:18.678851 STDOUT 2254] | super().close() |  
00:06:18.678889 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 105, in close |  
00:06:18.678891 STDOUT 2254] | self.flush() |  
00:06:18.678894 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 112, in flush |  
00:06:18.678917 STDOUT 2254] | with atomic_write.atomic_write(self.path) as f: |  
00:06:18.678918 STDOUT 2254] | File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ |  
00:06:18.678921 STDOUT 2254] | return next(self.gen) |  
00:06:18.678946 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/utils/atomic_write.py", line 50, in atomic_write |  
00:06:18.678949 STDOUT 2254] | with open(tmppath, "wb" if binary else "w") as file: |  
00:06:18.678952 STDOUT 2254] | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpsn3811ro/openaigym.episode_batch.0.7695.stats.json~' |  
00:06:18.679025 STDOUT 2254] | Exception ignored in: <function Monitor.__del__ at 0x7f068a825a70> |  
00:06:18.679029 STDOUT 2254] | Traceback (most recent call last): |  
00:06:18.679035 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 287, in __del__ |  
00:06:18.679086 STDOUT 2254] | self.close() |  
00:06:18.679089 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 180, in close |  
00:06:18.679094 STDOUT 2254] | self.stats_recorder.close() |  
00:06:18.679193 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/pfrl/wrappers/monitor.py", line 99, in close |  
00:06:19.013708 STDOUT 2254] | super().close() |  
00:06:19.013712 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 105, in close |  
00:06:19.013719 STDOUT 2254] | self.flush() |  
00:06:19.013810 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 112, in flush |  
00:06:19.013814 STDOUT 2254] | with atomic_write.atomic_write(self.path) as f: |  
00:06:19.013820 STDOUT 2254] | File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ |  
00:06:19.013865 STDOUT 2254] | return next(self.gen) |  
00:06:19.013867 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/utils/atomic_write.py", line 50, in atomic_write |  
00:06:19.013871 STDOUT 2254] | with open(tmppath, "wb" if binary else "w") as file: |  
00:06:19.013919 STDOUT 2254] | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpi_11hhxg/openaigym.episode_batch.1.7695.stats.json~' |  
00:06:19.013922 STDOUT 2254] | Error in atexit._run_exitfuncs: |  
00:06:19.013925 STDOUT 2254] | Traceback (most recent call last): |  
00:06:19.013968 STDOUT 2254] | File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ |  
00:06:19.013971 STDOUT 2254] | return next(self.gen) |  
00:06:19.013976 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/utils/atomic_write.py", line 50, in atomic_write |  
00:06:19.014031 STDOUT 2254] | with open(tmppath, "wb" if binary else "w") as file: |  
00:06:19.014034 STDOUT 2254] | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpu9hvcsn1/openaigym.episode_batch.2.7695.stats.json~' |  
00:06:19.014038 STDOUT 2254] | Exception ignored in: <function Monitor.__del__ at 0x7f068a825a70> |  
00:06:19.014067 STDOUT 2254] | Traceback (most recent call last): |  
00:06:19.014069 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 287, in __del__ |  
00:06:19.014072 STDOUT 2254] | self.close() |  
00:06:19.014096 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 180, in close |  
00:06:19.014098 STDOUT 2254] | self.stats_recorder.close() |  
00:06:19.014101 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/pfrl/wrappers/monitor.py", line 99, in close |  
00:06:19.014158 STDOUT 2254] | super().close() |  
00:06:19.014160 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 105, in close |  
00:06:19.014166 STDOUT 2254] | self.flush() |  
00:06:19.014219 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 112, in flush |  
00:06:19.014222 STDOUT 2254] | with atomic_write.atomic_write(self.path) as f: |  
00:06:19.014227 STDOUT 2254] | File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ |  
00:06:19.014344 STDOUT 2254] | return next(self.gen) |  
00:06:19.014351 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/utils/atomic_write.py", line 50, in atomic_write |  
00:06:19.348248 STDOUT 2254] | with open(tmppath, "wb" if binary else "w") as file: |  
00:06:19.348312 STDOUT 2254] | FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpu9hvcsn1/openaigym.episode_batch.2.7695.stats.json~' |  
00:06:19.348337 STDOUT 2254] | Exception ignored in: <function Monitor.__del__ at 0x7f068a825a70> |  
00:06:19.348344 STDOUT 2254] | Traceback (most recent call last): |  
00:06:19.348401 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 287, in __del__ |  
00:06:19.348403 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitor.py", line 180, in close |  
00:06:19.348407 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/pfrl/wrappers/monitor.py", line 99, in close |  
00:06:19.348467 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 105, in close |  
00:06:19.348475 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/wrappers/monitoring/stats_recorder.py", line 112, in flush |  
00:06:19.348484 STDOUT 2254] | File "/opt/conda/lib/python3.7/contextlib.py", line 112, in __enter__ |  
00:06:19.348571 STDOUT 2254] | File "/opt/conda/lib/python3.7/site-packages/gym/utils/atomic_write.py", line 50, in atomic_write |  
00:06:19.348576 STDOUT 2254] | NameError: name 'open' is not defined


created time in 13 days

PullRequestReviewEvent

pull request commentpfnet/pfrl

[BUGFIX] use functools.partial instead of lambda

Thanks for your PR! Now it looks fine (compared with https://github.com/pfnet/pfrl/issues/156#issuecomment-914225388):

python examples/atari/train_ppo_ale.py --gpu -1
Output files are saved in results/f8a561ae627d31138ac6fe975e06bfbfe20ee345-1c0dd107-67bc6343
make_env called with idx: 0
make_env called with idx: 1
make_env called with idx: 2
make_env called with idx: 3
make_env called with idx: 4
make_env called with idx: 5
make_env called with idx: 6
make_env called with idx: 7
Observation space Box(4, 84, 84)
Action space Discrete(4)
make_env called with idx: 0
make_env called with idx: 1
make_env called with idx: 2
make_env called with idx: 3
make_env called with idx: 4
make_env called with idx: 5
make_env called with idx: 6
make_env called with idx: 7
make_env called with idx: 0
make_env called with idx: 1
make_env called with idx: 2
make_env called with idx: 3
make_env called with idx: 4
make_env called with idx: 5
make_env called with idx: 6
make_env called with idx: 7
zhouzypaul

comment created time in 13 days

pull request commentpfnet/pfrl

[BUGFIX] use functools.partial instead of lambda

/test

zhouzypaul

comment created time in 13 days

issue commentpfnet/pfrl

[BUG] lazy execution issue with lambda

Oh, this is definitely a bug that affects performance. Thanks for pointing it out! I can reproduce:

diff --git a/examples/atari/train_ppo_ale.py b/examples/atari/train_ppo_ale.py
index aa6e107..11d8a17 100644
--- a/examples/atari/train_ppo_ale.py
+++ b/examples/atari/train_ppo_ale.py
@@ -174,6 +174,7 @@ def main():
     print("Output files are saved in {}".format(args.outdir))

     def make_env(idx, test):
+        print(f"make_env called with idx: {idx}")
         # Use different random seeds for train and test envs
         process_seed = int(process_seeds[idx])
         env_seed = 2 ** 32 - 1 - process_seed if test else process_seed
python examples/atari/train_ppo_ale.py --gpu -1
Output files are saved in results/13eb97e0e87096517935edba7673b448b2bec64f-2da9143a-67bc6343
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
Observation space Box(4, 84, 84)
Action space Discrete(4)
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
make_env called with idx: 7
zhouzypaul

comment created time in 13 days

startedassimp/assimp

started time in 14 days

pull request commentpfnet/pfrl

[WIP] Snapshot for preemption

Thanks for your PR! It is really good to have better resumability.

General comments on resumability

First, let me summarize what I think need to be done to achieve resumability. Please comment if I miss something. I checked the points supported by this PR.

Things that need to be snapshotted for resumability besides randomness:

  • [ ] Agent
    • [x] Model (Agent.save)
    • [x] Optimizer (Agent.save)
    • [x] Replay buffer
    • [ ] recurrent states (only when recurrent models are used)*
    • [ ] any other internal states of Agent (e.g. PPO holds its own dataset)
    • [ ] statistics
  • [ ] Env
    • [ ] Env's internal state*
  • [ ] Experiment record
    • [x] max_score
    • [x] steps
    • [ ] episodes
    • [ ] past evaluation results (to reproduce scores.txt)

* indicates things needed only when you resume a half-way training episode.

RNG-related things that need to be snapshotted for complete resumability:

  • [ ] Env's internal RNG states
  • [ ] Module-level RNG states of torch, random, and numpy
  • [ ] Any other random states used in code (gym.spaces has its own random state, for example)

This is a large list, and it would be a tough task to support all of it. I think it is ok to start supporting only part of it if

  • it is tested and confirmed that the snapshotting functionality works as users expect for some tasks, and
  • its limitations are clarified so that users can know when it won't work as expected.

Specific comments on this PR

  • max_score is saved separately, but I think it is better to load scores.txt to restore all the evaluation information.
  • Snapshotting after every checkpoint_freq steps is not always desirable since it would consume time and storage mostly due to replay buffer. It should be optional.
  • How fast is it to save and load a large replay buffer e.g. with 1 million transitions each of which is 10KB? (this is almost what you would expect when you run DQN for Atari, so by running the experiments suggested below you can see it too).
  • Can you run experiments to demonstrate this PR works in non-trivial settings? My suggestion is:
    • Run python examples/atari/reproduction/train_dqn.py --env SpaceInvadersNoFrameskip-v4 --steps 10000000 (which takes <1day with a single GPU, a single CPU, and 14GB CPU RAM) with snapshots saved. Run with five random seeds: --seed 0/1/2/3/4 since variance among runs is high.
    • Resume from 5000000 steps to 10000000 steps for each seed.
    • Compare they roughly match.
knshnb

comment created time in 18 days

startedenriccorona/SMPLicit

started time in 20 days

issue commentpfnet/pfrl

recurrent examples fails with "TypeError: pad_sequence(): sequences must be tuple of Tensors, not Tensor"

self.train_recurrent_states contains the necessary information, h_t and c_t in the case of LSTM, for inference at next step t+1. DQN does not store older states as they are no longer needed for inference.

If you need past recurrent states e.g. for updating the model, you can use recurrent states stored in the replay buffer https://github.com/pfnet/pfrl/blob/7b0c7e938ba2c0c56a941c766c68635d0dad43c8/pfrl/agents/dqn.py#L532-L542 .

apersonnaz

comment created time in a month

issue commentpfnet/pfrl

recurrent examples fails with "TypeError: pad_sequence(): sequences must be tuple of Tensors, not Tensor"

Thanks for reporting this. I was able to reproduce it. python examples/atari/train_ppo_ale.py --recurrent works fine with torch==1.8.1 but fails with torch==1.9.0. This should be fixed, but as a workaround you can use torch<1.9.0.

$ python examples/atari/train_ppo_ale.py --recurrent
Output files are saved in results/7b0c7e938ba2c0c56a941c766c68635d0dad43c8-00000000-7d123117
Observation space Box(0, 255, (4, 84, 84), uint8)
Action space Discrete(4)
INFO:pfrl.experiments.train_agent_batch:Saved the agent to results/7b0c7e938ba2c0c56a941c766c68635d0dad43c8-00000000-7d123117/0_except
Traceback (most recent call last):
  File "/home/fujita/pfrl/examples/atari/train_ppo_ale.py", line 333, in <module>
    main()
  File "/home/fujita/pfrl/examples/atari/train_ppo_ale.py", line 316, in main
    experiments.train_agent_batch_with_evaluation(
  File "/home/fujita/pfrl/pfrl/experiments/train_agent_batch.py", line 247, in train_agent_batch_with_evaluation
    eval_stats_history = train_agent_batch(
  File "/home/fujita/pfrl/pfrl/experiments/train_agent_batch.py", line 67, in train_agent_batch
    actions = agent.batch_act(obss)
  File "/home/fujita/pfrl/pfrl/agents/ppo.py", line 678, in batch_act
    return self._batch_act_train(batch_obs)
  File "/home/fujita/pfrl/pfrl/agents/ppo.py", line 731, in _batch_act_train
    ) = one_step_forward(
  File "/home/fujita/pfrl/pfrl/utils/recurrent.py", line 149, in one_step_forward
    pack = pack_one_step_batch_as_sequences(batch_input)
  File "/home/fujita/pfrl/pfrl/utils/recurrent.py", line 125, in pack_one_step_batch_as_sequences
    return nn.utils.rnn.pack_sequence(xs[:, None])
  File "/home/fujita/.local/share/virtualenvs/pfrl-pL7Y2GAq/lib/python3.9/site-packages/torch/nn/utils/rnn.py", line 398, in pack_sequence
    return pack_padded_sequence(pad_sequence(sequences), lengths, enforce_sorted=enforce_sorted)
  File "/home/fujita/.local/share/virtualenvs/pfrl-pL7Y2GAq/lib/python3.9/site-packages/torch/nn/utils/rnn.py", line 363, in pad_sequence
    return torch._C._nn.pad_sequence(sequences, batch_first, padding_value)
TypeError: pad_sequence(): argument 'sequences' (position 1) must be tuple of Tensors, not Tensor
apersonnaz

comment created time in a month

startedgoogle/mediapipe

started time in a month

startedhijkzzz/pymarl2

started time in a month

startedytrock/THuman2.0-Dataset

started time in a month

startednkolot/SPIN

started time in a month