Add type hints to APIs

[sethmlarson]

This will allow us to overcome the issue with query_params in that it cloaks the signature of each API function when it comes to query params. This means that IDEs will be able to show the full set of parameters for each API. Also nowadays Python libraries are moving towards type checking and the community will start desiring this functionality. It's nice when things work well out of the box. :)

Type Stubs / Python 2 compatible inline type hints

• Using type stubs means we don't have to break Python 2 support as inline type hints are only possible in 3.6+
• Python 2 compatible inline type hints are tough to parse since they aren't right next to the actual parameter, I've found them to be harder to read. They also would contribute to bloat of the existing .py files.

Interesting Places to Look

Anything that's not generated

All of these APIs were typed by hand, would appreciate reviews here the most.

Generated APIs

• Most of the APIs (anything under client/ and _async/client/) aren't very interesting beyond the following:
  • Everything is typed with Any unless I explicitly know what the type should be.
  • Any API that uses HEAD as an HTTP method returns bool because it's guaranteed to by Transport.
  • All of the parameters that are available on every request (format, human, filter_path, request_timeout, etc)
    have an appropriate type hint applied to them unless they're also used by the API (in the case of format).
  • Why Optional[Any] instead of Any? They aren't the same.

Aiohttp Catch 22

Since we have aiohttp and yarl as optional dependencies we enter a catch-22 situation when it comes to mypy.

• If we add # type: ignore to the line with import aiohttp then there is no problem when aiohttp isn't installed (because mypy can't find the type hints) but when it is installed mypy will then complain with "You don't need that # type: ignore comment" because aiohttp ships its own type hints and mypy isn't detecting any problems with them.
• If we don't add # type: ignore to the import we have the reversed problem where there's no issue when aiohttp is imported but now mypy will complain that no type hints can be found when using --strict mode. (As users should!)

Solved this by adding _async/_extra_imports.py which basically imports aiohttp and yarl and adds a file-wide # type: ignore which mypy will never complain about. Feels like a work-around but tbh I'm unsure there's any way around this without requiring users to configure their mypy individually which I'd prefer not to require.

Requests gets around this unfortunate situation by being distributed within typeshed so mypy is always able to find type hints regardless of whether the package is installed or not. Verified this by deleting it's typeshed entry and the errors started occuring for requests as well.

Added AsyncConnection

Basically an abstract class that makes perform_request() and close async functions. If in the future
there's another async HTTP connection class or if users want to define their own.

Type hints for AsyncGenerators

An async generator can be written like so:

async def async_gen():
    yield 0  # This yield makes this a generator

but within a pyi file typically you don't include a function body which means we'll be missing the yield

async def async_gen() -> AsyncGenerator[None, int]: ...

but mypy then complains that calls to async for x in async_gen() have type Coroutine. Instead we should do this:

def async_gen() -> AsyncGenerator[None, int]: ... 

now the type signature matches exactly what the function is, a non-async function that returns an async generator.

External Project Type Checking Tests

Under test_elasticsearch/test_types I have two files, one async and one sync which import and exercise the APIs
to see what it feels like as an external user of the library when using mypy type checking. I run with [async] installed
against both async and sync and then uninstall aiohttp and run mypy against the sync with --strict enabled to ensure
we're not leaking type errors to our users.

TODO

• Add typing to AIOHttpConnection.__init__

Closes #736

[briancurtin]

Would those aiohttp/yarl cases be handled by something like the following in a mypy.ini?

[aiohttp.*,yarl.*]
ignore_missing_imports = True

That way if hints are found they're used, but if they're not they're not.

[sethmlarson]

@briancurtin That would work but every user would then have to make that configuration themselves which is a bummer.

[briancurtin]

@briancurtin That would work but every user would then have to make that configuration themselves which is a bummer.

That seems a level deeper than I'm familiar with this going, though maybe I just misunderstand some of the layering here. If you're a consumer of elasticsearch, are typing issues inside the library actually exposed to consumers to where they'd need this? I've had to # type: ignore or the mypy.ini settings on things I directly use, but haven't seen problems with dependencies internal to those dependencies be exposed.

[sethmlarson]

@briancurtin Yeah that's the case for mypy, especially with --strict/--follow-imports mode which many projects use. I created a few files that use the APIs (look under test_elasticsearch/test_types/...) and run mypy --strict on them to ensure we're not leaking type errors. Even without --strict/--follow-imports mode we still leak a type error when the aiohttp library isn't installed which is the default mode the library is installed without the work-around. :/

Users would either have to use the above config you provided or use --no-warn-unused-ignores/--ignore-missing-imports to mypy to completely silence type errors.

[sethmlarson]

Data point: I've integrated this branch into a project using Eland which is partially type-checked and received no issues from elasticsearch-py when using mypy --strict.

[sethmlarson]

I rebased this branch on master and collapsed everything into three commits. The first two are deeply reviewable since they're written by hand but the last one should be high-level reviewable since it's all generated code.

[hub-cap]

Thanks a lot for this effort. Its great to see a project that is typed and python here. I don't have anything to add other than thank you and I will be taking these ideas back to my teams python projects as well. Keep up the great work!