Skip to content

[refactor] Move output files to a common results/ folder #3829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Apr 29, 2020

Conversation

ervteng
Copy link
Contributor

@ervteng ervteng commented Apr 23, 2020

Proposed change(s)

This PR changes the artifact output directory to this structure:

results/ 
    {run_id}/
        configuration.yaml - the configuration (after CLI overrides) used for this run.
        {behavior_name}/
            Tensorboard files
            snapshot.ckpt (and .pb) - model checkpoints and intermediate output files
        {behavior_name}.csv - CSV log
        {behavior_name}.nn - Barracuda Output
    run_logs/
        timers.json - timers file

In addition, we now output the RunOptions object as a YAML file configuration.yaml. When combined with PR #3815, the resulting YAML can then be used with mlagents-learn to re-run the same run again.

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

https://docs.google.com/document/d/1uyk5JVNevfWy2DmqVHy19lcDzyQCNppPxkPOgJcQ7z4/edit#heading=h.5cr9b5kpfi98

Types of change(s)

  • Bug fix
  • New feature
  • Code refactor
  • Breaking change
  • Documentation update
  • Other (please describe)

Checklist

  • Added tests that prove my fix is effective or that my feature works
  • Updated the changelog (if applicable)
  • Updated the documentation (if applicable)
  • Updated the migration guide (if applicable)

Other comments

@ervteng ervteng requested review from andrewcoh and harperj April 23, 2020 01:59
@ervteng ervteng marked this pull request as ready for review April 23, 2020 03:40
@ervteng ervteng requested a review from anupam-142857 April 23, 2020 03:40
@ervteng
Copy link
Contributor Author

ervteng commented Apr 23, 2020

@anupambhatnagar - Added you to the review for visibility as it will require a corresponding change to the CI pipeline.

@harperj
Copy link
Contributor

harperj commented Apr 23, 2020

@ervteng you're right that this will break the daily CI runs. I think there are 3 places in the CI we'll need to update:

  • results reporter
  • script for updating NN files
  • the runner script's barracuda inference logic

One of us could take a stab if that would be useful.

Unrelatedly, I think the structure you listed in the PR description is a little off -- all files should be nested under the run-id folder. I'd also suggest we might just get rid of the run_logs folder and move those files into the general run-id folder.

@ervteng
Copy link
Contributor Author

ervteng commented Apr 23, 2020

@harperj I've actually already made the CI changes in this PR: https://github.com/Unity-Technologies/ml-agents-cloud/pull/105 but definitely it needs a looksee from one of you to make sure I didn't miss anything. Seems to work when I tested it.

I think getting rid of the run-log (summaries) in the design doc could work, as there will only be two files in there (the timers.json and eventually the Player.Log). The only time it will make sense to have a separate run-log is when running many environments, as each will have their own Player.log (so Player-1.log, Player-2.log, etc.) and clutter up the {run-id} directory.

Ervin Teng and others added 5 commits April 27, 2020 18:37
Since the default Player.log path would be overwritten on subsequent
runs, we should keep the Unity Player logs in the results folder
for a training run.  This change uses the -logFile CLI option to the
Unity Player to set the path.
@ervteng ervteng merged commit 3fee8b9 into master Apr 29, 2020
@delete-merged-branch delete-merged-branch bot deleted the develop-results-folder branch April 29, 2020 22:27
@github-actions github-actions bot locked as resolved and limited conversation to collaborators May 14, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants