ppai/weather: add weather forecasting sample (#8345)

* ppai/weather-foreacasting: add weather sample

* remove temporary test model

* remove test scripts

* install local packages before testing

* update datasets

* add local packages to test requirements

* split notebook and tests

* add predictions notebook and tests

* add missing requirements

* update notebooks

* update test names

* fix tests

* enable vertex test

* copy file to directory

* update dependencies

* ignore checkpoints directory

* Simplify boolean test

Co-authored-by: Dan Lee <[email protected]>

* Improve wording

Co-authored-by: Dan Lee <[email protected]>

* Improve wording

Co-authored-by: Dan Lee <[email protected]>

* rename go to graph_objects

Co-authored-by: gcf-merge-on-green[bot] <60162190+gcf-merge-on-green[bot]@users.noreply.github.com>
Co-authored-by: Dan Lee <[email protected]>

Author: 3 people

people-and-planet-ai/README.md

@@ -2,8 +2,8 @@
 
 ## 🦏 [Wildlife Insights -- _image-classification_](image-classification)
 
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)
-
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/image-classification/README.ipynb)
+>
 > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/hUzODH3uGg0/0.jpg)](https://youtu.be/hUzODH3uGg0)
 
 This model is trained to recognize animal species from
@@ -20,10 +20,12 @@ pictures.
 [AutoML]: https://cloud.google.com/vertex-ai/docs/beginner/beginners-guide
 [Vertex AI]: https://cloud.google.com/vertex-ai
 
-## 🗺 [Global Fishing Watch -- _timeseries-classification_](timeseries-classification)
+---
 
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/timeseries-classification/README.ipynb)
+## 🗺 [Global Fishing Watch -- _timeseries-classification_](timeseries-classification)
 
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/timeseries-classification/README.ipynb)
+>
 > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/LnEhSVEJUuY/0.jpg)](https://youtu.be/LnEhSVEJUuY)
 
 This model is trained to classify if a ship is fishing or not every hour from their
@@ -41,10 +43,12 @@ location data.
 [Keras]: https://keras.io
 [Vertex AI]: https://cloud.google.com/vertex-ai
 
-## 🏭 [Coal Plant Predictions -- _geospatial-classification_](geospatial-classification)
+---
 
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/geospatial-classification/README.ipynb)
+## 🏭 [Coal Plant Predictions -- _geospatial-classification_](geospatial-classification)
 
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/geospatial-classification/README.ipynb)
+>
 > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/8amFK7T_n30/0.jpg)](https://youtu.be/8amFK7T_n30)
 
 This model uses satellite data to predict if a coal plant is turned on and producing carbon emissions. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
@@ -60,10 +64,12 @@ This model uses satellite data to predict if a coal plant is turned on and produ
 [TensorFlow]: https://www.tensorflow.org/
 [Vertex AI]: https://cloud.google.com/vertex-ai
 
-## 🌍 [Land cover classification -- _image segmentation_](land-cover-classification)
+---
 
-[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/land-cover-classification/README.ipynb)
+## 🌍 [Land cover classification -- _image segmentation_](land-cover-classification)
 
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/land-cover-classification/README.ipynb)
+>
 > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/zImQf91ffFo/0.jpg)](https://youtu.be/zImQf91ffFo)
 
 This model uses satellite data to classify what is on Earth. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
@@ -81,3 +87,22 @@ This model uses satellite data to classify what is on Earth. The satellite data
 [Earth Engine]: https://earthengine.google.com/
 [TensorFlow]: https://www.tensorflow.org/
 [Vertex AI]: https://cloud.google.com/vertex-ai
+
+---
+
+## 🌦 Weather forecasting -- _timeseries regression_
+
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/README.ipynb)
+<!-- > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/TODO/0.jpg)](https://youtu.be/TODO) -->
+
+This model uses satellite data to forecast precipitation for the next 2 and 6 hours. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
+
+* **Model**: 2D Fully Convolutional Network in [PyTorch]
+* **Creating datasets**: [Sentinel-2] satellite data and [ESA WorldCover] from [Earth Engine] with [Dataflow]
+* **Training the model**: [PyTorch] in [Vertex AI]
+* **Getting predictions**: [PyTorch] locally
+
+[Dataflow]: https://cloud.google.com/dataflow
+[Earth Engine]: https://earthengine.google.com/
+[PyTorch]: https://pytorch.org/
+[Vertex AI]: https://cloud.google.com/vertex-ai

people-and-planet-ai/conftest.py

@@ -14,18 +14,19 @@
 
 from __future__ import annotations
 
-from datetime import datetime
-import multiprocessing
 import os
 import platform
 import re
 import subprocess
-from unittest import mock
+import sys
+import textwrap
 import uuid
 from collections.abc import Callable, Iterable
+from datetime import datetime
+from unittest import mock
 
-from google.cloud import storage
 import pytest
+from google.cloud import storage
 
 
 @pytest.fixture(scope="session")
@@ -202,9 +203,9 @@ def run_notebook(
     skip_shell_commands: bool = False,
     until_end: bool = False,
 ) -> None:
+    import nbformat
     from nbclient.client import NotebookClient
     from nbclient.exceptions import CellExecutionError
-    import nbformat
 
     def notebook_filter_section(
         start: str,
@@ -262,13 +263,7 @@ def notebook_filter_section(
             cmd = "pass"
             cell["source"] = shell_command_re.sub(cmd, cell["source"])
         else:
-            cmd = [
-                "import subprocess",
-                "_cmd = f'''\\1'''",
-                "print(f'>> {_cmd}')",
-                "subprocess.run(_cmd, shell=True, check=True)",
-            ]
-            cell["source"] = shell_command_re.sub("\n".join(cmd), cell["source"])
+            cell["source"] = shell_command_re.sub(r"_run(f'''\1''')", cell["source"])
 
         # Apply variable substitutions.
         for regex, new_value in compiled_substitutions:
@@ -278,8 +273,38 @@ def notebook_filter_section(
         for old, new in replace.items():
             cell["source"] = cell["source"].replace(old, new)
 
+        # Clear outputs.
+        cell["outputs"] = []
+
     # Prepend the prelude cell.
-    nb.cells = [nbformat.v4.new_code_cell(prelude)] + nb.cells
+    prelude_src = textwrap.dedent(
+        """\
+        def _run(cmd):
+            import subprocess as _sp
+            import sys as _sys
+            _p = _sp.run(cmd, shell=True, stdout=_sp.PIPE, stderr=_sp.PIPE)
+            _stdout = _p.stdout.decode('utf-8').strip()
+            _stderr = _p.stderr.decode('utf-8').strip()
+            if _stdout:
+                print(f'➜ !{cmd}')
+                print(_stdout)
+            if _stderr:
+                print(f'➜ !{cmd}', file=_sys.stderr)
+                print(_stderr, file=_sys.stderr)
+            if _p.returncode:
+                raise RuntimeError('\\n'.join([
+                    f"Command returned non-zero exit status {_p.returncode}.",
+                    f"-------- command --------",
+                    f"{cmd}",
+                    f"-------- stderr --------",
+                    f"{_stderr}",
+                    f"-------- stdout --------",
+                    f"{_stdout}",
+                ]))
+        """
+        + prelude
+    )
+    nb.cells = [nbformat.v4.new_code_cell(prelude_src)] + nb.cells
 
     # Run the notebook.
     error = ""
@@ -289,7 +314,16 @@ def notebook_filter_section(
     except CellExecutionError as e:
         # Remove colors and other escape characters to make it easier to read in the logs.
         #   https://stackoverflow.com/a/33925425
-        error = re.sub(r"(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]", "", str(e))
+        color_chars = re.compile(r"(\x9B|\x1B\[)[0-?]*[ -\/]*[@-~]")
+        error = color_chars.sub("", str(e))
+        for cell in nb.cells:
+            if cell["cell_type"] != "code":
+                continue
+            for output in cell["outputs"]:
+                if output.get("name") == "stdout":
+                    print(color_chars.sub("", output["text"]))
+                elif output.get("name") == "stderr":
+                    print(color_chars.sub("", output["text"]), file=sys.stderr)
 
     if error:
         raise RuntimeError(
@@ -305,6 +339,8 @@ def run_notebook_parallel(
     replace: dict[str, str] = {},
     skip_shell_commands: bool = False,
 ) -> None:
+    import multiprocessing
+
     args = [
         {
             "ipynb_file": ipynb_file,

people-and-planet-ai/weather-forecasting/.gitignore

@@ -0,0 +1,7 @@
+**/*.egg-info
+**/build
+**/dist
+checkpoints/
+data/
+data-training/
+model/

people-and-planet-ai/weather-forecasting/README.md

@@ -0,0 +1,16 @@
+## 🌦 Weather forecasting -- _timeseries regression_
+
+> [![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/GoogleCloudPlatform/python-docs-samples/blob/main/people-and-planet-ai/weather-forecasting/README.ipynb)
+<!-- > [Watch the video in YouTube<br> ![thumbnail](http://img.youtube.com/vi/TODO/0.jpg)](https://youtu.be/TODO) -->
+
+This model uses satellite data to forecast precipitation for the next 2 and 6 hours. The satellite data comes from [Google Earth Engine.](https://earthengine.google.com/)
+
+* **Model**: 2D Fully Convolutional Network in [PyTorch]
+* **Creating datasets**: [Sentinel-2] satellite data and [ESA WorldCover] from [Earth Engine] with [Dataflow]
+* **Training the model**: [PyTorch] in [Vertex AI]
+* **Getting predictions**: [PyTorch] locally
+
+[Dataflow]: https://cloud.google.com/dataflow
+[Earth Engine]: https://earthengine.google.com/
+[PyTorch]: https://pytorch.org/
+[Vertex AI]: https://cloud.google.com/vertex-ai