add grpcio grpcio-status to support spark connect

[xieshuaihu]

Describe your changes

add grpcio and grpcio_status package in pyspark-notebook.

Before this pr, the pyspark-notebook:spark-3.5.0 will throw exception when connect to spark connect

Checklist (especially for first-time contributors)

• I have performed a self-review of my code
• If it is a core feature, I have added thorough tests
• I will try not to use force-push to make the review process easier for reviewers
• I have updated the documentation for significant changes

[mathbunnyru]

I sorted install entries and fixed package name, it should be grpcio-status.

[mathbunnyru]

Pyspark mentions these packages are required for Spark Connect, so it seems to be reasonable to install them.
https://spark.apache.org/docs/latest/api/python/getting_started/install.html#dependencies

@bjornjorgensen may I ask you to take a look here?

[mathbunnyru]

Please, update https://github.com/jupyter/docker-stacks/blob/main/docs/using/selecting.md#jupyterpyspark-notebook with newly added packages.

[xieshuaihu]

@mathbunnyru docs is updated

[bjornjorgensen]

It seams like we need to add versions and googleapis-common-protos

grpcio	>=1.48,<1.57	Required for Spark Connect
grpcio-status	>=1.48,<1.57	Required for Spark Connect
googleapis-common-protos	==1.56.4	Required for Spark Connect

[xieshuaihu]

@bjornjorgensen grpcio and grpcio-status versions are added. There is no need to dep googleapis-common-protos directly which is a traverse dep of grpcio-status

[mathbunnyru]

Please, update this file, to fix the tests: https://github.com/jupyter/docker-stacks/blob/main/tests/base-notebook/test_packages.py#L68

[xieshuaihu]

@mathbunnyru thanks for your tips

[mathbunnyru]

LGTM

@xieshuaihu a small tip for the future: try to keep lists sorted, if they are already sorted and you add new items to them.

[xieshuaihu]

Sorry for that, I didn't know the lists sorted. And Thanks again for your patience and tips

[mathbunnyru]

@xieshuaihu please, fix the build failure.
To build images locally, you need to run make build/<somestack>, starting with docker-stacks-foundation (so you have the up-to-date version) and then building images in the hierarchy till pyspark-notebook.

[xieshuaihu]

I am solving the deps conflict

[xieshuaihu]

@mathbunnyru I inspected pyspark setup.py, the required grpcio version is >=1.56, so i removed the version
https://github.com/apache/spark/blob/branch-3.5/python/setup.py#L135

[xieshuaihu]

It seams like we need to add versions and googleapis-common-protos

grpcio >=1.48,<1.57 Required for Spark Connect
grpcio-status >=1.48,<1.57 Required for Spark Connect
googleapis-common-protos ==1.56.4 Required for Spark Connect

It sees like this doc is out of date
https://github.com/apache/spark/blob/branch-3.5/python/setup.py#L135

[mathbunnyru]

@mathbunnyru I inspected pyspark setup.py, the required grpcio version is >=1.56, so i removed the version https://github.com/apache/spark/blob/branch-3.5/python/setup.py#L135

Thanks, it makes sense.

[xieshuaihu]

I don't know why the build failure again...

[xieshuaihu]

I inspected this build error, It seems like test-sparklyr in all-spark-notebook was failed. I don't know sparklry, help is expected!

[mathbunnyru]

This most likely has nothing to do with your PR.
Unfortunately, some tests seem to be flaky 😢
I restarted the failed builds.

[mathbunnyru]

Restart helped, merging this PR.

[mathbunnyru]

@xieshuaihu images have been updated, could you please test, that there is no error now?

[xieshuaihu]

Thanks very much, @mathbunnyru . I will test it today.

[xieshuaihu]

@xieshuaihu images have been updated, could you please test, that there is no error now?

I pulled the latest image, and run the following code, it works.

from pyspark.sql import SparkSession
spark = SparkSession.builder.remote("sc://host:port/;user_id=test").getOrCreate()
spark.sql("show databases").show()