Skip to content

add episodessampler #51

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 9, 2023
Merged

add episodessampler #51

merged 5 commits into from
Aug 9, 2023

Conversation

HenriDeh
Copy link
Member

@HenriDeh HenriDeh commented Aug 9, 2023

This PR adds a new sampler that will sample all the episodes in the Trajectory, even the truncated one (the first episode being typically the only one that's truncated in a limited size buffer).

The decision to sample all episodes is due to two reasons:

  1. Algorithms that use whole episode sampling are typically on-policy algorithms (eg. PPO, TRPO) which means that they use the entirety of the buffered transitions then discard them. As such, there is no reason to sample a subset of the buffer.
  2. There is no simple way to sample episodes without replacement from the buffer. As long as no algorithm expressly requires to subset entire Episodes, I don't see a compelling reason to add this functionality.

@test length(b[2][:state]) == 6
@test !haskey(b[1], :action)
end
end
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
end
end

@jeremiahpslewis jeremiahpslewis merged commit 0e76ebb into main Aug 9, 2023
@jeremiahpslewis jeremiahpslewis deleted the episodesampler branch August 9, 2023 14:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants