Merge branch 'main' into update_datatable

Author: LucaMarconato

.pre-commit-config.yaml

@@ -17,10 +17,12 @@ repos:
       rev: 23.10.1
       hooks:
           - id: black
+            exclude: .*.zarr/
     - repo: https://github.com/pre-commit/mirrors-prettier
       rev: v3.0.3
       hooks:
           - id: prettier
+            exclude: .*.zarr/
     - repo: https://github.com/asottile/blacken-docs
       rev: 1.16.0
       hooks:
@@ -45,11 +47,16 @@ repos:
       hooks:
           - id: detect-private-key
           - id: check-ast
+            exclude: .*.zarr/
           - id: end-of-file-fixer
+            exclude: .*.zarr/
           - id: mixed-line-ending
             args: [--fix=lf]
+            exclude: .*.zarr/
           - id: trailing-whitespace
+            exclude: .*.zarr/
           - id: check-case-conflict
+            exclude: .*.zarr/
     - repo: https://github.com/PyCQA/autoflake
       rev: v2.2.1
       hooks:

_static/img/speed.png

@@ -148,5 +148,17 @@ Many different technologies can be visualised out-of-the-box. See below for a co
 
                 notebooks/examples/technology_cosmx.ipynb
 
+    .. grid-item::
+
+        .. container:: custom-card
+
+            .. image:: _static/img/speed.png
+                :target: notebooks/examples/speed_up_illustration.html
+
+            .. toctree::
+                :maxdepth: 1
+
+                notebooks/examples/speed_up_illustration.ipynb
+
 
 ```

notebooks.md

@@ -14,24 +14,35 @@ This directory contains notebooks that operate on lightweight datasets.
 -   Each notebook covers a particular aspect of the storage specification and ~~all the~~ _the main (work in progress)_ edge cases of the specification are covered in at least one of the notebooks.
 -   All the notebooks are run every 24h (work in progress, automatic run temporarily disabled) against the `main` branch of the `spatialdata` repository. Each notebook creates a dataset, writes it to disk, reloads it in memory, rewrites it to disk to check for consistency, reloads it again in memory and plots it.
 -   The disk storage is committed to GitHub so that the output of each daily run is associated to a commit, the commit message is "autorun: storage format; spatialdata from <commit hash> <optional (commit tag)>". Examples of commit messages are:
-    -   `autorun: storage format; spatialdata from al29fak`
+    -   `autorun: storage format; spatialdata from al29fak (v0.0.12.q020ke.dev)`
     -   `autorun: storage format; spatialdata from fa096da (v0.0.12)`
 -   The `.zarr` data produced by every run is available in the current directory, in the commit corresponding to the run.
--   The data is also [uploaded to S3](https://refined-github-html-preview.kidonng.workers.dev/scverse/spatialdata-notebooks/raw/dev_notebooks/notebooks/developers_resources/storage_format/index.html), both as Zarr directories and as zipped files.
+-   The data is also [uploaded to S3](https://refined-github-html-preview.kidonng.workers.dev/scverse/spatialdata-notebooks/raw/main/notebooks/developers_resources/storage_format/index.html), both as Zarr directories and as zipped files (upload to S3 temporarily disable, please download the data from the GitHub repository).
 
 ## How to use this repository
 
 Practically, a third party tool (e.g. R reader, format converter, JavaScript data visualizer, etc.) that runs correctly on the lightweight datasets from this repository, should be guaranteed to run correctly on any SpatialData dataset.
 
 We recommend the following.
 
--   Implement your readers on the data from the latest run available (look for the latest commit with message `autorun: storage format; ...`).
+-   Implement your readers on the data from the latest commit available
 -   Set up an automated test (e.g. daily) that gets the latest converted data (you can use a `git pull` or download the data from S3) and runs your code on it.
--   If your reader fails, you can inspect the corresponding commit in this repository to see what has changed in the storage specification; in particular, you may find useful to compare different commits using the GitHub compare function, accessible with the following syntax: https://github.com/scverse/spatialdata-notebooks/compare/267adb1..5847084
+-   If your reader fails, you can inspect the corresponding commit in this repository to see what has changed in the storage specification; in particular:
+    -   you may find useful to compare different commits using the GitHub compare function, accessible with the following syntax: https://github.com/scverse/spatialdata-notebooks/compare/f55f89a..d8e4d55;
+    -   you may want to compare the latest commit with one of the commits using the spaitaldata version (or a closely related dev version), of the one that you used when implementing the readers; the versions are specified in the commit messages as described before.
+
+Furthermore, you can look at the on-disk versioning. Each element specifies the encoding that has been used for writing; you can find this in the .zattrs file ([example, see the end](https://github.com/scverse/spatialdata-notebooks/blob/main/notebooks/developers_resources/storage_format/transformation_affine.zarr/points/blobs_points/.zattrs)).
+A global version for the `SpatialData` object that gets bumped every time something changes at the top-level (or every time the encoding of an element changes) is not available yet, but [its implementation is tracked here](https://github.com/scverse/spatialdata/issues/356).
 
 ## Important technical notes
 
 -   The most crucial part of the metadata is stored, for each spatial element, in the `.zattr` file. [Example](transformation_identity.zarr/images/blobs_image/.zattrs).
 -   The `zmetadata` in the root folder stores redundant information and is used for storage systems that do not support `ls` operations (e.g. S3). [Example](transformation_identity.zarr/zmetadata).
 -   Please keep in mind that the data that we generate daily are produced against the latest `main` and not the latest release. This means that in the event of a format change (which should anyway happen less and less frequently as the frameworks become more mature), this does not immediately translate into a bug for the user. In fact, the user will still be using the latest release version for a while, giving time to developers to update the tools before the users are affected.
 -   When the format will become more mature we will provide converters between previous versions of the format. Luckily, heavy data like images and labels are stable from NGFF v0.4, therefore the converters will mostly perform lightweight conversions of the metadata and relatively small conversions of the geometries.
+
+# TODOs:
+
+-   [ ] re-enable daily runs, and remove "work in progress" notice from the readme.
+-   [ ] merge the transformations notebooks into one
+-   [ ] re-enable upload to S3, and remove "work in progress" notice from the readme.

notebooks/developers_resources/storage_format/Readme.md

@@ -36,7 +36,20 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "094509a1-86fb-4aa6-b920-11e34ed43e1c",
+   "id": "4804890f-bd05-4d59-b83c-939f8429cf16",
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import warnings\n",
+    "\n",
+    "# warnings clutter the diffs of notebooks\n",
+    "warnings.filterwarnings(\"ignore\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "id": "f57d4446-f975-4b44-a78c-0583258b9b92",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -46,7 +59,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "332c9c62-7451-4d8e-ba9e-13c12131c312",
+   "id": "8e37f99e-8bd7-45d1-8b12-b29f8d2dcf81",
    "metadata": {},
    "outputs": [],
    "source": [
@@ -113,7 +126,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.12"
+   "version": "3.10.13"
   }
  },
  "nbformat": 4,

notebooks/developers_resources/storage_format/__template__.ipynb

@@ -1,3 +1,4 @@
+import logging
 import os
 import shutil
 import tempfile
@@ -8,6 +9,34 @@
 from spatialdata._io._utils import _are_directories_identical
 
 
+class DisableLogger:
+    """
+    Context manager that temporarily disables specific logging messages.
+
+    Parameters
+    ----------
+    level
+        The logging level to disable. It can be a numeric level or a string representation
+        of the level as defined in the logging module.
+
+    Examples
+    --------
+    from logging import INFO
+
+    with DisableLogger(INFO):
+        logging.info('This will not be logged')
+    """
+
+    def __init__(self, level):
+        self.level = level
+
+    def __enter__(self):
+        logging.disable(self.level)
+
+    def __exit__(self, exit_type, exit_value, exit_traceback):
+        logging.disable(logging.NOTSET)
+
+
 def delete_old_data(name: str) -> None:
     """
     Safely delete an old Zarr store, it if exists.
@@ -58,7 +87,12 @@ def write_sdata_and_check_consistency(sdata: sd.SpatialData, name: str) -> None:
     with tempfile.TemporaryDirectory() as tmpdir:
         sdata2 = sd.read_zarr(f1)
         f2 = os.path.join(tmpdir, f"{name}2.zarr")
-        sdata2.write(f2)
+        with DisableLogger(logging.INFO):
+            # remove the message
+            #    INFO     The Zarr file used for backing will now change from multiple_elements.zarr to
+            #             /tmp/tmp15sd47gc/multiple_elements2.zarr
+            # which otherwise appears in the git diff
+            sdata2.write(f2)
         assert _are_directories_identical(f1, f2)
 
     shutil.make_archive(f1, "zip", f1)

notebooks/developers_resources/storage_format/io_utils.py

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.ipynb

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/.zgroup

@@ -90,4 +90,4 @@
             "version": "0.4"
         }
     ]
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/.zgroup

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_image/.zattrs

@@ -22,4 +22,4 @@
         64
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_image/.zgroup

@@ -116,4 +116,4 @@
             "version": "0.4"
         }
     ]
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_image/0/.zarray

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_multiscale_image/.zattrs

@@ -22,4 +22,4 @@
         64
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_multiscale_image/.zgroup

@@ -22,4 +22,4 @@
         32
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_multiscale_image/0/.zarray

@@ -22,4 +22,4 @@
         16
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_multiscale_image/1/.zarray

@@ -3,4 +3,4 @@
         "blobs_labels",
         "blobs_multiscale_labels"
     ]
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/images/blobs_multiscale_image/2/.zarray

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/.zattrs

@@ -67,4 +67,4 @@
             "version": "0.4"
         }
     ]
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/.zgroup

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_labels/.zattrs

@@ -20,4 +20,4 @@
         64
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_labels/.zgroup

@@ -91,4 +91,4 @@
             "version": "0.4"
         }
     ]
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_labels/0/.zarray

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_multiscale_labels/.zattrs

@@ -20,4 +20,4 @@
         64
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_multiscale_labels/.zgroup

@@ -20,4 +20,4 @@
         32
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_multiscale_labels/0/.zarray

@@ -20,4 +20,4 @@
         16
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_multiscale_labels/1/.zarray

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/labels/blobs_multiscale_labels/2/.zarray

@@ -41,4 +41,4 @@
         "instance_key": "instance_id",
         "version": "0.1"
     }
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/points/.zgroup

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/points/blobs_points/.zattrs

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/points/blobs_points/.zgroup

@@ -43,4 +43,4 @@
         },
         "version": "0.1"
     }
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/points/blobs_points/points.parquet/part.0.parquet

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/shapes/.zgroup

@@ -18,4 +18,4 @@
         5
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/shapes/blobs_circles/.zattrs

@@ -20,4 +20,4 @@
         2
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/shapes/blobs_circles/.zgroup

@@ -18,4 +18,4 @@
         5
     ],
     "zarr_format": 2
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/shapes/blobs_circles/Index/.zarray

@@ -43,4 +43,4 @@
         },
         "version": "0.1"
     }
-}
+}

notebooks/developers_resources/storage_format/multiple_elements.zarr/shapes/blobs_circles/coords/.zarray

@@ -1,3 +1,3 @@
 {
     "zarr_format": 2
-}
+}