bpo-36871: Ensure method signature is used when asserting mock calls to a method #13261

tirkarthi · 2019-05-12T07:30:58Z

https://bugs.python.org/issue36871

gpshead · 2019-05-12T16:51:20Z

Lib/unittest/mock.py

    def _call_matcher(self, _call):
        """
        Given a call (or simply an (args, kwargs) tuple), return a
        comparison key suitable for matching with other calls.
        This is a best effort method which relies on the spec's signature,
        if available, or falls back on the arguments themselves.
        """
-        sig = self._spec_signature
+
+        if isinstance(_call, tuple) and len(_call) > 2:


Q: Is being of tuple type actually important, or is any sequence okay?
My assumption: tuple is easy to express and being an internal API we may be able to guarantee the type so this could be fine.

Yes, any sequence can be used. It should be a two element sequence args and kwargs or three element sequence in case of name being present. But most examples and tests construct call which inherits from tuple or tuple as a recommended one.

>>> from unittest.mock import Mock >>> m = Mock() >>> m(1) <Mock name='mock()' id='4554182024'> >>> m.assert_has_calls([[(1,), {}]])

This feels like it would be sensible to just state and ensure that these are only ever _Call objects.

@tirkarthi - thoughts on my comment here ^^

Though this is an internal method there is a note on the docstring of _call_matcher that it could be a tuple. It's not specified in docs that this could be a tuple. I haven't seen a usecase of it in other projects too. There is a test for this usecase but it's not using autospec so it didn't fail with my PR I guess. Relevant test using tuples

cpython/Lib/unittest/test/testmock/testmock.py

Line 1265 in 9fe42b4

((3, 4),), ((), {'a': 3}),

Given a call (or simply an (args, kwargs) tuple), return a comparison key suitable for matching with other calls.

There could be a below test that passes with checking _call to be a tuple.

def test_assert_has_calls_tuple(self): class Something: def __init__(self): pass def meth(self, a, b, c, d=None): pass m = create_autospec(Something) m.meth(1, 2, 3, 1) m.assert_has_calls([('meth', (1, 2, 3, 1), {})])

On having a strict check of only using _Call it will fail since it will go on to use the constructor signature.

./python.exe -m unittest Lib.unittest.test.testmock.testmock.MockTest.test_assert_has_calls_tuple F ====================================================================== FAIL: test_assert_has_calls_tuple (Lib.unittest.test.testmock.testmock.MockTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/test/testmock/testmock.py", line 1387, in test_assert_has_calls_tuple m.assert_has_calls([('meth', (1, 2, 3, 1), {})]) File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/mock.py", line 876, in assert_has_calls raise AssertionError( AssertionError: Calls not found. Expected: [('meth', (1, 2, 3, 1), {})] Actual: [call.meth(1, 2, 3, 1)]. ---------------------------------------------------------------------- Ran 1 test in 0.063s FAILED (failures=1)

I am leaning towards removing tuple to support only _Call since the error message is also not useful. We can perhaps change it in 3.9 to accept only _Call and then revert back if someone complains that it breaks their workflow given there is plenty of time. I am not sure about backporting it to 3.8 though since beta is out. Would also like to know thoughts from @voidspace since this was added in the original commit where mock was added.

Ah, if it being _Call-only causes problems then that's probably a good reason to leave it as it is..

gpshead · 2019-05-12T16:56:46Z

Lib/unittest/test/testmock/testmock.py

+        mock.assert_has_calls([call.meth(1, 2, 3, 1)])
+
+        mock = create_autospec(Something)
+        mock.Foo().meth1(1, 2)


You cover two cases here mocked.meth(...) and mocked.Foo().meth1(...) that this PR fixes the behavior of. Good! Lets also cover the cases that I believe were already working for completeness sake:

mocked().meth(...) and mocked().Foo().meth1(...).

Thanks, I have added it. I also made a mistake where Something.Foo was not having a constructor and Something.Foo.meth1 was using Something.Foo with signature (*args, **kwargs). So I have added a single argument constructor for this to make sure I am using correct signature of meth1.

I also noted there was a logical mistake in my iteration where I was using self._mock_children.get('Foo') and was again calling Foo._mock_children.get('Foo') inside the loop. Due to above constructor mistake accepting (*args, **kwargs) it was not caught. There is also an internal implementation where unless an attribute is accessed spec stores _SpecState object for the key and that also now has a test.

Do all the things you've discovered now have test coverage? If so, can you point me at the tests?

gpshead · 2019-05-12T16:57:22Z

Lib/unittest/test/testmock/testmock.py

+
+                def meth1(self, a, b): pass
+
+        mock = create_autospec(Something)


given we are testing the mock module, lets not assign to a local name mock. perhaps call this mocked or mock_something.

...or mock_.

…te was not accessed and add tests.

tirkarthi · 2019-06-23T15:42:58Z

@mariocj89 Would be helpful to have your review of this.

The main thing I feel is that this lacks tests. I have added some and fixed some bugs in my own test. Also if there is a good way to get to the mock's spec from it's that has parenthesis wrapping around it for nested calls like Foo().Bar().Spam then the logic to get the correct spec in the helper function can be simplified.

cjw296 · 2019-06-24T08:14:28Z

What tests do you feel this lacks? What's stopping them being added?

…uctor signature for check

tirkarthi · 2019-06-24T18:39:56Z

I added some more tests. My general concern was that getting to correct spec for nested calls since I was doing text manipulation to generate the list of names to lookup. I have used different signatures for all the methods and checked for non existent attributes in nested calls. I think it's good now.

Please review the changes. Thanks.

cjw296 · 2019-07-03T06:42:33Z

Lib/unittest/test/testmock/testmock.py

+        mock_class = create_autospec(Something)
+
+        for m in [mock_class, mock_class()]:
+            m.meth(1, 2, 3, d=1)


I'm confused: for the mock_class case, why is this expected to work? meth isn't a classmethod...

I too had the same confusion :) It seems for both class and instance the signature skips self as in the below example. There is some note in docs at https://docs.python.org/3/library/unittest.mock.html#create-autospec but I am also little surprised at this.

If a class is used as a spec then the return value of the mock (the instance of the class) will have the same spec. You can use a class as the spec for an instance object by passing instance=True. The returned mock will only be callable if instances of the mock are callable.

from unittest.mock import * import inspect class A: def meth1(self, a): pass a = create_autospec(A) a.meth1(1) print(inspect.signature(a.meth1)) print(a.mock_calls) # Use as an object obj = a() obj.meth1(1) print(inspect.signature(obj.meth1)) print(obj.mock_calls)

./python.exe /tmp/bar.py (a) [call.meth1(1)] (a) [call.meth1(1)]

Similar test :

cpython/Lib/unittest/test/testmock/testhelpers.py

Lines 533 to 543 in 0f6f73f

def test_method_calls(self):

class Sub(SomeClass):

attr = SomeClass()

mock = create_autospec(Sub)

mock.one(1, 2)

mock.two()

mock.three(3)

expected = [call.one(1, 2), call.two(), call.three(3)]

self.assertEqual(mock.method_calls, expected)

It's unrelated to this PR, but this just feels like a bug to me. If you've autospecced from a class, the mock shouldn't have things that behave like classmethods if those things aren't classmethods on the object the spec is from.

The "similar test" you've included is just as confusing: why should the mock have one, two or three attributes when the original does not? @voidspace?

one, two and three are methods of SomeClass that is inherited by Sub

Ah! That makes more sense :-)

cjw296 · 2019-07-03T06:45:34Z

Lib/unittest/mock.py

+    def _get_call_signature_from_name(self, name):
+        """
+        If there is no name like the case for call check against functions
+        with tuples then return the self's spec signature.


I can't really follow what this sentence is saying, could it be reworded?

I have reworded the comment. Feel free to suggest if it can be rephrased better. An explanation using examples would be as below but I am not sure of using examples in the code comment.

If an assertion is made against a method like obj.meth1.assert_has_calls([call(1)] here call(1) object has no name to lookup and hence just return obj.meth1._spec_signature.

If an assertion is made against the obj like obj.assert_has_calls([call.meth1(1)]) here call.meth1(1) will have name 'meth1' that should be used to return signature of obj.meth1 unlike the bug where obj signature was returned.

Thanks

cjw296

Only nitpicks left from me :-)

cjw296 · 2019-07-03T07:48:23Z

Lib/unittest/mock.py

+        * If call objects are asserted against a method/function like obj.meth1
+        then there could be no name for the call object to lookup. Hence just
+        return the spec_signature of the method/function being asserted against.
+        * If the name is not empty then remove () and split by '.' to get
        list of names to iterate through the children until a potential


much clearer, thanks!

ned-deily · 2019-08-29T06:05:42Z

This PR seems to have stalled. Any core developer want to follow up on merging it?

miss-islington · 2019-08-29T06:09:21Z

Thanks @tirkarthi for the PR, and @cjw296 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

miss-islington · 2019-08-29T06:09:21Z

Thanks @tirkarthi for the PR, and @cjw296 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2019-09-25T04:06:55Z

Thanks @tirkarthi for the PR, and @cjw296 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

miss-islington · 2019-09-25T04:06:55Z

Thanks @tirkarthi for the PR, and @cjw296 for merging it 🌮🎉.. I'm working now to backport this PR to: 3.7.
🐍🍒⛏🤖

gpshead · 2019-09-25T04:10:23Z

PR #15578 was the backport to 3.8.
PR #15577 was the backport to 3.7.

tirkarthi added 2 commits May 12, 2019 12:56

Fix call_matcher for mock when using methods

acfe960

Add NEWS entry

4788924

the-knights-who-say-ni added the CLA signed label May 12, 2019

bedevere-bot added the awaiting review label May 12, 2019

Use None check and convert doctest to unittest

5e9e4e6

csabella requested review from cjw296, lisroach and gpshead May 12, 2019 09:24

gpshead reviewed May 12, 2019

View reviewed changes

Use better name for mock in tests. Handle _SpecState when the attribu…

ec2daf5

…te was not accessed and add tests.

Use reset_mock instead of reinitialization. Change inner class constr…

01aac71

…uctor signature for check

cjw296 reviewed Jul 3, 2019

View reviewed changes

cjw296 approved these changes Jul 3, 2019

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting review labels Jul 3, 2019

Reword comment regarding call object lookup logic

67c9299

cjw296 reviewed Jul 3, 2019

View reviewed changes

cjw296 merged commit c961278 into python:master Aug 29, 2019

bedevere-bot removed the awaiting merge label Aug 29, 2019

cjw296 added needs backport to 3.7 labels Aug 29, 2019

sfreilich mentioned this pull request Sep 25, 2019

bpo-36871: Handle spec errors in assert_has_calls #16005

Merged

gpshead removed the needs backport to 3.7 label Sep 25, 2019

gpshead added needs backport to 3.7 and removed needs backport to 3.8 labels Sep 25, 2019

gpshead removed needs backport to 3.7 labels Sep 25, 2019


		def meth1(self, a, b): pass

		mock = create_autospec(Something)

	def test_method_calls(self):
	class Sub(SomeClass):
	attr = SomeClass()

	mock = create_autospec(Sub)
	mock.one(1, 2)
	mock.two()
	mock.three(3)

	expected = [call.one(1, 2), call.two(), call.three(3)]
	self.assertEqual(mock.method_calls, expected)

Uh oh!

bpo-36871: Ensure method signature is used when asserting mock calls to a method #13261

bpo-36871: Ensure method signature is used when asserting mock calls to a method #13261

Uh oh!

Conversation

tirkarthi commented May 12, 2019 • edited by bedevere-bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tirkarthi commented Jun 23, 2019

Uh oh!

cjw296 commented Jun 24, 2019

Uh oh!

tirkarthi commented Jun 24, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tirkarthi Jul 3, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cjw296 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ned-deily commented Aug 29, 2019

Uh oh!

miss-islington commented Aug 29, 2019

Uh oh!

miss-islington commented Aug 29, 2019

Uh oh!

miss-islington commented Sep 25, 2019

Uh oh!

miss-islington commented Sep 25, 2019

Uh oh!

gpshead commented Sep 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

tirkarthi commented May 12, 2019 •

edited by bedevere-bot

Loading

tirkarthi Jul 3, 2019 •

edited

Loading

gpshead commented Sep 25, 2019 •

edited

Loading