Skip to content

Reduce size of installed dependencies by slimming #191

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 27 commits into from
Jun 15, 2018
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
91d80c3
replace all double backslashes
dee-me-tree-or-love May 14, 2018
76a92b0
add the slimmen logic
dee-me-tree-or-love May 14, 2018
7b7c429
adding test
dee-me-tree-or-love May 14, 2018
4362153
added a few more tests
dee-me-tree-or-love May 14, 2018
8389e2b
fix useless-escape eslint error
dee-me-tree-or-love May 14, 2018
b2d50fa
triple equal and one time folder path preparation
dee-me-tree-or-love May 15, 2018
f500910
prettier --write formatting
dee-me-tree-or-love May 15, 2018
82c1088
remove 1. from readme
dee-me-tree-or-love May 16, 2018
780af23
improved tests and a few bug fixes
dee-me-tree-or-love May 16, 2018
81e2663
better evaluate slim option in js
dee-me-tree-or-love May 16, 2018
c406b5f
add self to readme
dee-me-tree-or-love May 16, 2018
4302c17
prettier code without doublequotes
dee-me-tree-or-love May 16, 2018
1f9ae75
added a default in index.js + a dockerizePip disclaimer in README
dee-me-tree-or-love May 17, 2018
eeee343
moved slim option handling to apply globally + added failing pipenv test
dee-me-tree-or-love May 18, 2018
e121d17
split code to a separate file + unix-compatibility check + passing te…
dee-me-tree-or-love May 18, 2018
f12d600
fix the prettier issus
dee-me-tree-or-love May 22, 2018
474b823
added handling slim as array of removal patterns
dee-me-tree-or-love May 22, 2018
d3a6dc8
prettier fix
dee-me-tree-or-love May 22, 2018
26216d8
adding custom slim patterns tests
dee-me-tree-or-love May 22, 2018
83b170c
adjusted custom removed directories, added tests
dee-me-tree-or-love May 22, 2018
671cc72
update README and remove tests
dee-me-tree-or-love May 22, 2018
2230ae4
prettier code fix
dee-me-tree-or-love May 22, 2018
97212b2
README line fix
dee-me-tree-or-love May 22, 2018
89c46ba
remove forgotten log
dee-me-tree-or-love May 22, 2018
c71be73
Merge branch 'master' into master
dee-me-tree-or-love May 23, 2018
33433aa
Merge branch 'master' into master
dee-me-tree-or-love Jun 9, 2018
e6a4273
Merge branch 'master' into master
dschep Jun 15, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,15 @@ try:
except ImportError:
pass
```
### Slimmen Package
To remove the tests, information and caches from the installed packages,
enable the `slim` option. This will: 1. `strip` the `.so` files, remove `__pycache__`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the 1. if you're not actually gonna have a numbered list 😉

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, yeah, thanks a lot! will fix it up 😄

directories, remove `tests` directories remove `dist-info` directories.
```yaml
custom:
pythonRequirements:
slim: true
```

## Omitting Packages
You can omit a package from deployment with the `noDeploy` option. Note that
Expand Down
26 changes: 25 additions & 1 deletion lib/pip.js
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,12 @@ function installRequirements(
}
cmdOptions.push(dockerImage);
cmdOptions.push(...pipCmd);

// If enabled slimming, strip out the caches, tests and dist-infos
if (options.slim === true) {
const slimCmd = getSlimPackageCommands(options, targetRequirementsFolder);
cmdOptions.push(...slimCmd);
}
} else {
cmd = pipCmd[0];
cmdOptions = pipCmd.slice(1);
Expand Down Expand Up @@ -147,11 +153,29 @@ function installRequirements(
*/
function dockerPathForWin(options, path) {
if (process.platform === 'win32' && options.dockerizePip) {
return path.replace('\\', '/');
return path.replace(/\\/g, '/');
}
return path;
}

/**
* get commands to slimmen the installed requirements
* @param {Object} options
* @param {string} targetRequirementsFolder
* @return {Array}
*/
function getSlimPackageCommands(options, targetRequirementsFolder) {
const folderPath = dockerPathForWin(options, targetRequirementsFolder);
const stripCmd = [
`&& find ${folderPath} -name "*.so" -exec strip {} ;`,
`&& find ${folderPath} -name "*.py[c|o]" -exec rm -rf {} +`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to use -exec rm instead of find's -delete option?

Copy link
Contributor Author

@dee-me-tree-or-love dee-me-tree-or-love May 16, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest I have never used t go for -delete - it felt natural to go with rm {} + since it's always there and as far as I know + makes it not span the redundant processes 🤔

Thanks for pointing it out - I have to consider opting for -delete!

It is available at the alpine docker image sooo, maybe I should run some experiments to check which performs better?
Do you personally prefer -delete 🙂 ?

`&& find ${folderPath} -type d -name "*tests*" -exec rm -rf {} +`,
`&& find ${folderPath} -type d -name "__pycache__*" -exec rm -rf {} +`,
`&& find ${folderPath} -type d -name "*.dist-info*" -exec rm -rf {} +`
];
return stripCmd;
}

/** create a filtered requirements.txt without anything from noDeploy
* @param {string} source requirements
* @param {string} target requirements where results are written
Expand Down
47 changes: 47 additions & 0 deletions test.bats
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,15 @@ teardown() {
ls puck/.requirements.zip puck/unzip_requirements.py
}

@test "py3.6 can package flask with zip & slim & dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
! uname -sm|grep Linux || groups|grep docker || id -u|egrep '^0$' || skip "can't dockerize on linux if not root & not in docker group"
sls --dockerizePip=true --zip=true --slim=true package
unzip .serverless/sls-py-req-test.zip -d puck
ls puck/.requirements.zip puck/unzip_requirements.py
}

@test "py3.6 can package flask with dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
Expand All @@ -67,6 +76,15 @@ teardown() {
ls puck/flask
}

@test "py3.6 can package flask with slim & dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
! uname -sm|grep Linux || groups|grep docker || id -u|egrep '^0$' || skip "can't dockerize on linux if not root & not in docker group"
sls --dockerizePip=true --slim=true package
unzip .serverless/sls-py-req-test.zip -d puck
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙌 Thanks for including tests too!!

Could you add this to also check that the unzipped services doesn't contain any pyc files?

test $(find . -name *.pyc | wc -l) -eq 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that's an awesome one, thanks, didn't know of this, will do now! ✌️

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should I put test $(find puck -name *.pyc | wc -l) -eq 0 maybe? so it runs inside of the puck folder?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup. My mistake.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dee-me-tree-or-love, @dschep Wouldn't you want to dump the .py and not .pyc?

ls puck/flask
}

@test "py3.6 uses cache with dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
Expand All @@ -76,6 +94,16 @@ teardown() {
ls .requirements-cache/http
}

@test "py3.6 uses cache with dockerizePip & slim option" {
cd tests/base
npm i $(npm pack ../..)
! uname -sm|grep Linux || groups|grep docker || id -u|egrep '^0$' || skip "can't dockerize on linux if not root & not in docker group"
perl -p -i'.bak' -e 's/(pythonRequirements:$)/\1\n pipCmdExtraArgs: ["--cache-dir", ".requirements-cache"]/' serverless.yml
sls --dockerizePip=true --slim=true package
ls .requirements-cache/http
}


@test "py2.7 can package flask with default options" {
cd tests/base
npm i $(npm pack ../..)
Expand Down Expand Up @@ -118,6 +146,15 @@ teardown() {
ls puck/.requirements.zip puck/unzip_requirements.py
}

@test "py2.7 can package flask with zip & slim & dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
! uname -sm|grep Linux || groups|grep docker || id -u|egrep '^0$' || skip "can't dockerize on linux if not root & not in docker group"
sls --dockerizePip=true --runtime=python2.7 --zip=true --slim=true package
unzip .serverless/sls-py-req-test.zip -d puck
ls puck/.requirements.zip puck/unzip_requirements.py
}

@test "py2.7 can package flask with dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
Expand All @@ -127,6 +164,16 @@ teardown() {
ls puck/flask
}

@test "py2.7 can package flask with slim & dockerizePip option" {
cd tests/base
npm i $(npm pack ../..)
! uname -sm|grep Linux || groups|grep docker || id -u|egrep '^0$' || skip "can't dockerize on linux if not root & not in docker group"
sls --dockerizePip=true --slim=true --runtime=python2.7 package
unzip .serverless/sls-py-req-test.zip -d puck
ls puck/flask
}


@test "pipenv py3.6 can package flask with default options" {
cd tests/pipenv
npm i $(npm pack ../..)
Expand Down
1 change: 1 addition & 0 deletions tests/base/serverless.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ custom:
pythonRequirements:
zip: ${opt:zip, self:custom.defaults.zip}
dockerizePip: ${opt:dockerizePip, self:custom.defaults.dockerizePip}
slim: ${opt:slim}
vendor: ${opt:vendor, ''}
defaults:
zip: false
Expand Down