Skip to content

PYTHON-2382 Destroy codec options struct in _cbson._element_to_dict #496

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

prashantmital
Copy link
Contributor

@prashantmital prashantmital commented Oct 6, 2020

Tested using a slightly modified version of the user-provided repro script:

import gc
from typing import Any
from pymongo import MongoClient
from bson.codec_options import CodecOptions
from bson.codec_options import TypeCodec, TypeRegistry


# %% Define method to analyse the memory
def memory_report(prefix):
    gc.collect()
    rr = {}

    for obj in gc.get_objects():

        tt = type(obj)
        if ('bson.codec_options.CodecOptions' in str(tt)):
            refs = gc.get_referrers(obj)
            print(tt)
            print(len(refs))
            # if len(refs) == 1:
            #     objgraph.show_refs([obj], filename='refs.png')
            rr[tt] = rr.get(tt, 0) + 1

    print(prefix + ' memory report:')
    for key in rr:
        nn = rr[key]
        print(f' {key}: {nn}')


# %% Create codec

class MyClass:
    pass


class MyCodec(TypeCodec):  # type: ignore

    @property
    def python_type(self) -> Any:
        return MyClass

    def transform_python(self, value: Any) -> Any:
        # print(f'transform_python: {value}')
        return value

    @property
    def bson_type(self) -> Any:
        return int

    def transform_bson(self, value: Any) -> Any:
        # print(f'transform_bson: {value}')
        return value * 2


codec = MyCodec()
type_registry = TypeRegistry([codec])
codec_options = CodecOptions(type_registry=type_registry)

client = MongoClient(replicaset='repl0')
database = client['my_database']
collection = database['my_collection'].with_options(codec_options)
collection.drop()

memory_report('before ')
for ii in range(10):
    collection.insert_one({'a': 2})
    cursor = collection.find({})
    for data in cursor:
        assert data['a'] == 4
memory_report('after ')

@peendebak
Copy link

@prashantmital Tested on my system and fixes the leak. Thanks!

Copy link
Member

@ShaneHarvey ShaneHarvey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. FYI I think this bug also occurs when inflating a RawBSONDocument.

@prashantmital prashantmital merged commit 594b211 into mongodb:master Oct 6, 2020
@prashantmital prashantmital deleted the PYTHON-2382/memory-leak-fix branch October 6, 2020 18:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants