Skip to content

Commit c6b0e6e

Browse files
wkliaowkliao
authored
Merge pull request #53 from wkliao/doc_v3
revise documentation
2 parents 4e2c331 + 1308eef commit c6b0e6e

File tree

14 files changed

+979
-582
lines changed

14 files changed

+979
-582
lines changed

‎README.md

Lines changed: 22 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,45 @@
1-
# PnetCDF python
1+
# PnetCDF for Python
22
![](https://img.shields.io/badge/python-v3.9-blue)
33
![](https://img.shields.io/badge/tests%20passed-49-brightgreen)
44
![](https://readthedocs.org/projects/pnetcdf-python/badge/?version=latest)
55

6-
PnetCDF-python is a Python interface to
7-
[PnetCDF](https://parallel-netcdf.github.io/), a high-performance parallel I/O
8-
library for accessing netCDF files.
9-
This package allows Python users to access netCDF data using the rich ecosystem
10-
of Python's scientific computing libraries, making it a valuable tool for
11-
applications that require parallel access to netCDF files.
6+
PnetCDF-Python is a Python interface to
7+
[PnetCDF](https://parallel-netcdf.github.io/), a high-performance I/O library
8+
for accessing netCDF files in parallel. It can provide MPI-based parallel
9+
python programs to achieve a scalable I/O performance.
1210

1311
### Software Dependencies
14-
* Python 3.9 or above
15-
* MPI libraries
16-
* PnetCDF [C library](https://github.com/Parallel-netCDF/PnetCDF), built with shared libraries.
17-
* Python library [mpi4py](https://mpi4py.readthedocs.io/en/stable/install.html)
18-
* Python library [numpy](http://www.numpy.org/)
12+
* Python 3.9 or later.
13+
* [numpy](http://www.numpy.org/) Python package.
14+
* MPI C library and Python package, [mpi4py](https://mpi4py.readthedocs.io/en/stable/install.html).
15+
* [PnetCDF C library](https://github.com/Parallel-netCDF/PnetCDF), built with shared libraries.
1916

2017
### Developer Installation
2118
* Clone this GitHub repository
22-
* Make sure the above dependent software are installed.
23-
* In addition, [Cython](http://cython.org/), [packaging](https://pypi.org/project/packaging/), [setuptools>=65](https://pypi.org/project/setuptools/) and [wheel](https://pypi.org/project/wheel/) are required for developer installation.
24-
* Set the environment variable `PNETCDF_DIR` to PnetCDF's installation path.
25-
* Make sure utility program `pnetcdf-config` is available in `$PNETCDF_DIR/bin`.
26-
* Run command below to install.
19+
* Required software for developer installation:
20+
+ The above mentioned dependent software are installed and additionally,
21+
+ [Cython](http://cython.org/), [packaging](https://pypi.org/project/packaging/), [setuptools>=65](https://pypi.org/project/setuptools/) and [wheel](https://pypi.org/project/wheel/).
22+
* Commands to install.
2723
```
28-
CC=/path/to/mpicc PNETCDF_DIR=/path/to/pnetcdf/dir pip install --no-build-isolation -e .
24+
export CC=/path/to/mpicc
25+
export PNETCDF_DIR=/path/to/pnetcdf/dir
26+
pip install --no-build-isolation -e .
2927
```
30-
* Testing
31-
+ Run command `"make check"` to test all the programs available in folders
32-
["test/"](./test) and ["examples/"](./examples) in parallel on 4 MPI
33-
processes.
34-
+ In addition, command `"make ptests"` runs the same tests using 3, 4, and 8
35-
MPI processes.
28+
* Testing -- Command `"make check"` tests all the programs available in folders
29+
["test/"](./test) and ["examples/"](./examples).
3630

3731
### Additional Resources
32+
* [Example python programs](./examples#pnetcdf-python-examples) available in
33+
folder [./examples](./examples).
3834
* PnetCDF-python [User Guide](https://pnetcdf-python.readthedocs.io/en/latest)
3935
* [Data objects](docs/pnetcdf_objects.md) in PnetCDF python programming
4036
* [Comparison](docs/nc4_vs_pnetcdf.md) of NetCDF4-python and PnetCDF-python
4137
* [PnetCDF project home page](https://parallel-netcdf.github.io)
42-
* [PnetCDF repository of C/Fortran library](https://github.com/Parallel-NetCDF/PnetCDF)
38+
* [PnetCDF of C/Fortran library repository](https://github.com/Parallel-NetCDF/PnetCDF)
4339

4440
### Developer Team
4541
* Youjia Li <<[email protected]>>
46-
* Wei-keng Liao <<[email protected]>> (Principle Investigator)
42+
* Wei-keng Liao <<[email protected]>>
4743

4844
### Acknowledgements
4945
Ongoing development and maintenance of PnetCDF-python is supported by the U.S.

‎docs/copyright.rst

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
================
2+
Copyright Statement
3+
================
4+
5+
::
6+
7+
Copyright (c) 2024 Northwestern University and Argonne National
8+
Laboratory All rights reserved.
9+
10+
Portions of this software were developed by the Unidata Program at the
11+
University Corporation for Atmospheric Research.
12+
13+
Access and use of this software shall impose the following obligations and
14+
understandings on the user. The user is granted the right, without any fee
15+
or cost, to use, copy, modify, alter, enhance and distribute this software,
16+
and any derivative works thereof, and its supporting documentation for any
17+
purpose whatsoever, provided that this entire notice appears in all copies
18+
of the software, derivative works and supporting documentation. Further,
19+
Northwestern University and Argonne National Laboratory request that the
20+
user credit Northwestern University and Argonne National Laboratory in any
21+
publications that result from the use of this software or in any product
22+
that includes this software. The names Northwestern University and Argonne
23+
National Laboratory, however, may not be used in any advertising or
24+
publicity to endorse or promote any products or commercial entity unless
25+
specific written permission is obtained from Northwestern University and
26+
Argonne National Laboratory. The user also understands that Northwestern
27+
University and Argonne National Laboratory are not obligated to provide the
28+
user with any support, consulting, training or assistance of any kind with
29+
regard to the use, operation and performance of this software nor to
30+
provide the user with any updates, revisions, new versions or "bug fixes."
31+
32+
THIS SOFTWARE IS PROVIDED BY NORTHWESTERN UNIVERSITY AND ARGONNE NATIONAL
33+
LABORATORY "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT
34+
NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A
35+
PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL NORTHWESTERN
36+
UNIVERSITY AND ARGONNE NATIONAL LABORATORY BE LIABLE FOR ANY SPECIAL,
37+
INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
38+
LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE
39+
OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE ACCESS,
40+
USE OR PERFORMANCE OF THIS SOFTWARE.
41+

‎docs/nc4_vs_pnetcdf.md

Lines changed: 217 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1-
# Difference between NetCDF4-python and PnetCDF-python
1+
# Comparison between PnetCDF-Python and NetCDF4-Python
2+
3+
Programming using [NetCDF4-Python](http://unidata.github.io/netcdf4-python/) and
4+
[PnetCDF-Python](https://pnetcdf-python.readthedocs.io) are very similar.
5+
Below lists some of the differences, including the file format support and
6+
operational modes.
27

38
* [Supported File Formats](#supported-file-formats)
49
* [Differences in Python Programming](#differences-in-python-programming)
10+
* [Define Mode and Data Mode](#define-mode-and-data-mode)
11+
* [Collective and Independent I/O Mode](#collective-and-independent-io-mode)
512
* [Blocking vs. Nonblocking APIs](#blocking-vs-nonblocking-apis)
613

714
---
@@ -59,6 +66,215 @@
5966
| ... ||
6067
| # close file<br>f.close() | ditto NetCDF4 |
6168

69+
---
70+
## Define Mode and Data Mode
71+
72+
In PnetCDF, an opened file is in either define mode or data mode. Switching
73+
between the modes is done by explicitly calling `"pnetcdf.File.enddef()"` and
74+
`"pnetcdf.File.redef()"`. NetCDF4-Python has no such mode switching
75+
requirement. The reason of PnetCDF enforcing such a requirement is to ensure
76+
the metadata consistency across all the MPI processes and keep the overhead of
77+
metadata synchronization small.
78+
79+
* Define mode
80+
+ When calling constructor of python class `"pnetcdf.File()"` to create a new
81+
file, the file is automatically put in the define mode. While in the
82+
define mode, the python program can create new dimensions, i.e. instances
83+
of class `"pnetcdf.Dimension"`, new variables, i.e. instances of class
84+
`"pnetcdf.Variable"`, and netCDF attributes. Modification of these data
85+
objects' metadata can only be done when the file is in the define mode.
86+
+ When opening an existing file, the opened file is automatically put in the
87+
data mode. To add or modify the metadata, a python program must call
88+
`"pnetcdf.File.redef()"`.
89+
90+
* Data mode
91+
+ Once the creation or modification of metadata is complete, the python
92+
program must call `"pnetcdf.File.enddef()"` to leave the define mode and
93+
enter the data mode.
94+
+ While an open file is in data mode, the python program can make read and
95+
write requests to that variables that have been created.
96+
97+
<ul>
98+
<li> A PnetCDF-Python example shows switching between define and data modes
99+
after creating a new file.</li>
100+
<li> <details>
101+
<summary>Example code fragment (click to expand)</summary>
102+
103+
```python
104+
import pnetcdf
105+
...
106+
# Create the file
107+
f = pnetcdf.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD)
108+
...
109+
# Define dimensions
110+
dim_y = f.def_dim("Y", 16)
111+
dim_x = f.def_dim("X", 32)
112+
113+
# Define a 2D variable of integer type
114+
var = f.def_var("grid", pnetcdf.NC_INT, (dim_y, dim_x))
115+
116+
# Add an attribute of string type to the variable
117+
var.str_att_name = "example attribute"
118+
119+
# Exit the define mode
120+
f.enddef()
121+
122+
# Write to a subarray of the variable, var
123+
var[4:8, 20:24] = buf
124+
125+
# Re-enter the define mode
126+
f.redef()
127+
128+
# Define a new 2D variable of float type
129+
var_flt = f.def_var("temperature", pnetcdf.NC_FLOAT, (dim_y, dim_x))
130+
131+
# Exit the define mode
132+
f.enddef()
133+
134+
# Write to a subarray of the variable, var_flt
135+
var_flt[0:4, 16:20] = buf_flt
136+
137+
# Close the file
138+
f.close()
139+
```
140+
</details></li>
141+
142+
<li> An example shows switching between define and data modes after opening an existing file.
143+
</li>
144+
<li> <details>
145+
<summary>Example code fragment (click to expand)</summary>
146+
147+
```python
148+
import pnetcdf
149+
...
150+
# Opening an existing file
151+
f = pnetcdf.File(filename, 'r', MPI.COMM_WORLD)
152+
...
153+
# get the python handler of variable named 'grid', a 2D variable of integer type
154+
var = f.variables['grid']
155+
156+
# Read the variable's attribute named "str_att_name"
157+
str_att = var.str_att_name
158+
159+
# Read a subarray of the variable, var
160+
r_buf = np.empty((4, 4), var.dtype)
161+
r_buf = var[4:8, 20:24]
162+
163+
# Re-enter the define mode
164+
f.redef()
165+
166+
# Define a new 2D variable of double type
167+
var_dbl = f.def_var("precipitation", pnetcdf.NC_DOUBLE, (dim_y, dim_x))
168+
169+
# Add an attribute of string type to the variable
170+
var_dbl.unit = "mm/s"
171+
172+
# Exit the define mode
173+
f.enddef()
174+
175+
# Write to a subarray of the variable, temperature
176+
var_dbl[0:4, 16:20] = buf_dbl
177+
178+
# Close the file
179+
f.close()
180+
```
181+
</details></li>
182+
</ul>
183+
184+
185+
---
186+
## Collective and Independent I/O Mode
187+
188+
The terminology of collective and independent I/O comes from MPI standard. A
189+
collective I/O function call requires all the MPI processes opening the same
190+
file to participate. On the other hand, an independent I/O function can be
191+
called by an MPI process independently from others.
192+
193+
For metadata I/O, both PnetCDF and NetCDF4 require the function calls to be
194+
collective.
195+
196+
* Mode Switch Mechanism
197+
+ PnetCDF-Python -- when a file is in the data mode, it can be put into
198+
either collective or independent I/O mode. The default mode is collective
199+
I/O mode. Switching to and exiting from the independent I/O mode is done
200+
by explicitly calling `"pnetcdf.File.begin_indep()"` and
201+
`"pnetcdf.File.end_indep()"`.
202+
203+
+ NetCDF4-Python -- collective and independent mode switching is done per
204+
variable basis. Switching mode is done by explicitly calling
205+
`"Variable.set_collective()"` before accessing the variable.
206+
For more information, see
207+
[NetCDF4-Python User Guide on Parallel I/O](https://unidata.github.io/netcdf4-python/#parallel-io)
208+
209+
<ul>
210+
<li> A PnetCDF-Python example shows switching between collective and
211+
independent I/O modes.</li>
212+
<li> <details>
213+
<summary>Example code fragment (click to expand)</summary>
214+
215+
```python
216+
import pnetcdf
217+
...
218+
# Create the file
219+
f = pnetcdf.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD)
220+
...
221+
# Metadata operations to define dimensions and variables
222+
...
223+
# Exit the define mode (by default, in the collective I/O mode)
224+
f.enddef()
225+
226+
# Write to variables collectively
227+
var_flt[start_y:end_y, start_x:end_x] = buf_flt
228+
var_dbl[start_y:end_y, start_x:end_x] = buf_dbl
229+
230+
# Leaving collective I/O mode and entering independent I/O mode
231+
f.begin_indep()
232+
233+
# Write to variables independently
234+
var_flt[start_y:end_y, start_x:end_x] = buf_flt
235+
var_dbl[start_y:end_y, start_x:end_x] = buf_dbl
236+
237+
# Close the file
238+
f.close()
239+
```
240+
</details></li>
241+
</ul>
242+
243+
<ul>
244+
<li> A NetCDF4-Python example shows switching between collective and
245+
independent I/O modes.</li>
246+
<li> <details>
247+
<summary>Example code fragment (click to expand)</summary>
248+
249+
```python
250+
import netCDF4
251+
...
252+
# Create the file
253+
f = netCDF4.File(filename, 'w', "NC_64BIT_DATA", MPI.COMM_WORLD, parallel=True)
254+
...
255+
# Metadata operations to define dimensions and variables
256+
...
257+
258+
# Write to variables collectively
259+
var_flt.set_collective(True)
260+
var_flt[start_y:end_y, start_x:end_x] = buf_flt
261+
262+
var_dbl.set_collective(True)
263+
var_dbl[start_y:end_y, start_x:end_x] = buf_dbl
264+
265+
# Write to variables independently
266+
var_flt.set_collective(False)
267+
var_flt[start_y:end_y, start_x:end_x] = buf_flt
268+
269+
var_dbl.set_collective(False)
270+
var_dbl[start_y:end_y, start_x:end_x] = buf_dbl
271+
272+
# Close the file
273+
f.close()
274+
```
275+
</details></li>
276+
</ul>
277+
62278
---
63279

64280
## Blocking vs Nonblocking APIs

‎docs/source/api/dimension_api.rst

Lines changed: 15 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,30 +1,30 @@
1-
.. currentmodule:: pnetcdf
21
==============
32
Dimension
43
==============
54

6-
Dimension defines the shape and structure of variables and stores
7-
coordinate data for multidimensional arrays. The ``Dimension`` object,
8-
which is also a key component of ``File`` class, provides an interface
9-
to access dimensions.
10-
11-
.. note::
12-
13-
``Dimension`` instances should be created using the :meth:`File.def_dim` method of a ``File`` instance,
14-
not using :meth:`Dimension.__init__` directly.
5+
Dimension defines the shape and structure of variables and stores coordinate
6+
data for multidimensional arrays. The ``Dimension`` object, which is also a key
7+
component of ``File`` class, provides an interface to access dimensions.
158

169
.. autoclass:: pnetcdf::Dimension
1710
:members: getfile, isunlimited
1811
:exclude-members: name, size
1912

20-
Dimension Attributes
21-
The following class members are read-only and should not be modified by the user.
13+
Read-only Python Attributes of Dimension Class
14+
The following class members are read-only and should not be modified by the
15+
user.
2216

2317
.. attribute:: name
2418

2519
String name of Dimension instance. This class member is read-only and
26-
should not be modified by the user. To rename a dimension, use :meth:`File.rename_dim` method.
20+
should not be modified by the user. To rename a dimension, use
21+
:meth:`File.rename_dim` method.
22+
23+
**Type:** `str`
2724

2825
.. attribute:: size
29-
30-
The current size of Dimension (calls ``len`` on Dimension instance).
26+
27+
The current size of Dimension (calls ``len`` on Dimension instance).
28+
29+
**Type:** `int`
30+

0 commit comments

Comments
 (0)