Skip to content

Autoformatting and documentation #33

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 17, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
[flake8]
# Ignore style and complexity
# E: style errors
# W: style warnings
# F401: module imported but unused
# F811: redefinition of unused `name` from line `N`
# F841: local variable assigned but never used
ignore = E, C, W, F401, F403, F811, F841, E402, I100, I101, D400
exclude =
helm-chart,
hooks,
setup.py,
statuspage,
versioneer.py
1 change: 0 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -23,4 +23,3 @@ var/
*.egg-info/
.installed.cfg
*.egg

21 changes: 21 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
repos:
- repo: https://github.com/asottile/reorder_python_imports
rev: v1.3.5
hooks:
- id: reorder-python-imports
language_version: python3.6
- repo: https://github.com/ambv/black
rev: 18.9b0
hooks:
- id: black
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.1.0
hooks:
- id: end-of-file-fixer
- id: check-json
- id: check-yaml
exclude: ^helm-chart/nbviewer/templates/
- id: check-case-conflict
- id: check-executables-have-shebangs
- id: requirements-txt-fixer
- id: flake8
36 changes: 36 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
language: python

python:
- 3.7
- 3.8

before_install:
- pip install --upgrade setuptools pip

install:
- pip install --editable .

script:
- python -m pytest -vvv nbresuse


jobs:
include:
- name: autoformatting check
python: 3.6
# NOTE: It does not suffice to override to: null, [], or [""]. Travis will
# fall back to the default if we do.
before_install: echo "Do nothing before install."
install: pip install pre-commit
script:
- pre-commit run --all-files
after_success: echo "Do nothing after success."
after_failure:
- |
echo "You can install pre-commit hooks to automatically run formatting"
echo "on each commit with:"
echo " pre-commit install"
echo "or you can run by hand on staged files with"
echo " pre-commit run"
echo "or after-the-fact on already committed files with"
echo " pre-commit run --all-files"
42 changes: 40 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
# Contributing

Contributions to nbresuse are highly welcome!
Contributions to NBResuse are highly welcome! As a [Jupyter](https://jupyter.org) project,
you can follow the [Jupyter contributor guide](https://jupyter.readthedocs.io/en/latest/contributor/content-contributor.html).

Make sure to also follow [Project Jupyter's Code of Conduct](https://github.com/jupyter/governance/blob/master/conduct/code_of_conduct.md)
for a friendly and welcoming collaborative environment.

## Development set up

Expand Down Expand Up @@ -51,4 +55,38 @@ We recommend using [pipenv](https://docs.pipenv.org/) to make development easier

```bash
MEM_LIMIT=$(expr 128 \* 1024 \* 1024) jupyter notebook
```
```

8. NBResuse has adopted automatic code formatting so you shouldn't
need to worry too much about your code style.
As long as your code is valid,
the pre-commit hook should take care of how it should look. Here is how to set up pre-commit hooks for automatic code formatting, etc.

```bash
pre-commit install
```

You can also invoke the pre-commit hook manually at any time with

```bash
pre-commit run
```

which should run any autoformatting on your code
and tell you about any errors it couldn't fix automatically.
You may also install [black integration](https://github.com/ambv/black#editor-integration)
into your text editor to format code automatically.

If you have already committed files before setting up the pre-commit
hook with `pre-commit install`, you can fix everything up using
`pre-commit run --all-files`. You need to make the fixing commit
yourself after that.

9. It's a good idea to write tests to exercise any new features,
or that trigger any bugs that you have fixed to catch regressions. `pytest` is used to run the test suite. You can run the tests with:

```bash
python -m pytest -vvv nbresuse
```

in the repo directory.
28 changes: 20 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,21 @@
# nbresuse
**[Installation](#installation)** |
**[Configuration](#configuration)** |
**[Resources Displayed](#resources-displayed)** |
**[Contributing](#contributing)**

# NBResuse

[![PyPI](https://img.shields.io/pypi/v/nbresuse.svg)](https://pypi.python.org/pypi/nbresuse)
[![PyPI](https://img.shields.io/pypi/l/nbresuse.svg)](https://pypi.python.org/pypi/nbresuse)
[![GitHub](https://img.shields.io/badge/issue_tracking-github-blue?logo=github)](https://github.com/yuvipanda/nbresuse/issues)
[![TravisCI build status](https://img.shields.io/travis/jupyter/nbviewer/master?logo=travis)](https://travis-ci.org/yuvipanda/nbresuse)
[![Build Status](https://dev.azure.com/tpaine154/jupyter/_apis/build/status/timkpaine.nbresuse?branchName=master)](https://dev.azure.com/tpaine154/jupyter/_build/latest?definitionId=17&branchName=master)
[![Coverage](https://img.shields.io/azure-devops/coverage/tpaine154/jupyter/17)](https://dev.azure.com/tpaine154/jupyter/_build?definitionId=17&_a=summary)
[![PyPI](https://img.shields.io/pypi/l/nbresuse.svg)](https://pypi.python.org/pypi/nbresuse)
[![PyPI](https://img.shields.io/pypi/v/nbresuse.svg)](https://pypi.python.org/pypi/nbresuse)



![Screenshot with memory limit](screenshot.png)

NB Resource Usage (nbresuse) is a small extension for Jupyter Notebooks that
NB Resource Usage (NBResuse) is a small extension for Jupyter Notebooks that
displays an indication of how much resources your current notebook server and
its children (kernels, terminals, etc) are using. This is displayed in the
main toolbar in the notebook itself, refreshing every 5s.
Expand Down Expand Up @@ -80,9 +87,14 @@ As a command line argument:
jupyter notebook --ResourceUseDisplay.track_cpu_percent=True
```

## Resources displayed
## Resources Displayed

Currently it only displays Memory usage (just RSS). Other metrics will be
Currently the server extension only reports memory usage (just RSS) and CPU usage. Other metrics will be
added in the future as needed.

The notebook extension currently doesn't show CPU usage.
The notebook extension currently doesn't show CPU usage, only memory usage.

## Contributing

If you would like to contribute to the project, please read the [`CONTRIBUTING.md`](CONTRIBUTING.md). The `CONTRIBUTING.md` file
explains how to set up a development installation and how to run the test suite.
44 changes: 24 additions & 20 deletions nbresuse/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import os

from tornado import ioloop
from traitlets import Bool, Float, Int, Union, default
from traitlets import Bool
from traitlets import default
from traitlets import Float
from traitlets import Int
from traitlets import Union
from traitlets.config import Configurable

from nbresuse.prometheus import PrometheusHandler
Expand All @@ -17,21 +21,21 @@ def _jupyter_server_extension_paths():
"""
Set up the server extension for collecting metrics
"""
return [{
'module': 'nbresuse',
}]
return [{"module": "nbresuse"}]


def _jupyter_nbextension_paths():
"""
Set up the notebook extension for displaying metrics
"""
return [{
"section": "notebook",
"dest": "nbresuse",
"src": "static",
"require": "nbresuse/main"
}]
return [
{
"section": "notebook",
"dest": "nbresuse",
"src": "static",
"require": "nbresuse/main",
}
]


class ResourceUseDisplay(Configurable):
Expand All @@ -49,7 +53,7 @@ class ResourceUseDisplay(Configurable):
we will start warning the user when they use (128 - (128 * 0.1)) MB.

Set to 0 to disable warning.
"""
""",
).tag(config=True)

mem_limit = Union(
Expand All @@ -62,18 +66,18 @@ class ResourceUseDisplay(Configurable):

Defaults to reading from the `MEM_LIMIT` environment variable. If
set to 0, the max memory available is displayed.
"""
""",
).tag(config=True)

@default('mem_limit')
@default("mem_limit")
def _mem_limit_default(self):
return int(os.environ.get('MEM_LIMIT', 0))
return int(os.environ.get("MEM_LIMIT", 0))

track_cpu_percent = Bool(
default_value=False,
help="""
Set to True in order to enable reporting of CPU usage statistics.
"""
""",
).tag(config=True)

cpu_warning_threshold = Float(
Expand All @@ -86,7 +90,7 @@ def _mem_limit_default(self):
we will start warning the user when they use (150 - (150 * 0.1)) %.

Set to 0 to disable warning.
"""
""",
).tag(config=True)

cpu_limit = Float(
Expand All @@ -98,19 +102,19 @@ def _mem_limit_default(self):

Defaults to reading from the `CPU_LIMIT` environment variable. If
set to 0, the total CPU count available is displayed.
"""
""",
).tag(config=True)

@default('cpu_limit')
@default("cpu_limit")
def _cpu_limit_default(self):
return float(os.environ.get('CPU_LIMIT', 0))
return float(os.environ.get("CPU_LIMIT", 0))


def load_jupyter_server_extension(nbapp):
"""
Called during notebook start
"""
resuseconfig = ResourceUseDisplay(parent=nbapp)
nbapp.web_app.settings['nbresuse_display_config'] = resuseconfig
nbapp.web_app.settings["nbresuse_display_config"] = resuseconfig
callback = ioloop.PeriodicCallback(PrometheusHandler(nbapp), 1000)
callback.start()
1 change: 0 additions & 1 deletion nbresuse/etc/nbextension.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,3 @@
"nbresuse/main": true
}
}

1 change: 0 additions & 1 deletion nbresuse/etc/serverextension.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,3 @@
}
}
}

11 changes: 3 additions & 8 deletions nbresuse/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,7 @@ def memory_metrics() -> MemoryMetrics:
rss = sum([p.memory_info().rss for p in all_processes])
virtual_memory = psutil.virtual_memory()

return MemoryMetrics(
rss,
virtual_memory.total
)
return MemoryMetrics(rss, virtual_memory.total)


def cpu_metrics() -> CPUMetrics:
Expand All @@ -39,9 +36,7 @@ def get_cpu_percent(p):
# about dead processes having no CPU usage
except BaseException:
return 0

cpu_percent = sum([get_cpu_percent(p) for p in all_processes])

return CPUMetrics(
cpu_count * 100.0,
cpu_percent
)
return CPUMetrics(cpu_count * 100.0, cpu_percent)
31 changes: 9 additions & 22 deletions nbresuse/prometheus.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,43 +2,30 @@
from prometheus_client import Gauge
from tornado import gen

from nbresuse.metrics import CPUMetrics, MemoryMetrics, cpu_metrics, memory_metrics
from nbresuse.metrics import cpu_metrics
from nbresuse.metrics import CPUMetrics
from nbresuse.metrics import memory_metrics
from nbresuse.metrics import MemoryMetrics

try:
# Traitlets >= 4.3.3
from traitlets import Callable
except ImportError:
from .utils import Callable

TOTAL_MEMORY_USAGE = Gauge(
'total_memory_usage',
'counter for total memory usage',
[]
)
TOTAL_MEMORY_USAGE = Gauge("total_memory_usage", "counter for total memory usage", [])

MAX_MEMORY_USAGE = Gauge(
'max_memory_usage',
'counter for max memory usage',
[]
)
MAX_MEMORY_USAGE = Gauge("max_memory_usage", "counter for max memory usage", [])

TOTAL_CPU_USAGE = Gauge(
'total_cpu_usage',
'counter for total cpu usage',
[]
)
TOTAL_CPU_USAGE = Gauge("total_cpu_usage", "counter for total cpu usage", [])

MAX_CPU_USAGE = Gauge(
'max_cpu_usage',
'counter for max cpu usage',
[]
)
MAX_CPU_USAGE = Gauge("max_cpu_usage", "counter for max cpu usage", [])


class PrometheusHandler(Callable):
def __init__(self, nbapp: NotebookApp):
super().__init__()
self.config = nbapp.web_app.settings['nbresuse_display_config']
self.config = nbapp.web_app.settings["nbresuse_display_config"]
self.session_manager = nbapp.session_manager

@gen.coroutine
Expand Down
Loading