Skip to content

ParallelImageCapture: Add continuous capture on espressif #5540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Nov 4, 2021

Conversation

jepler
Copy link

@jepler jepler commented Nov 3, 2021

By having a pair of buffers, the capture hardware can fill one buffer while
Python code (including displayio, etc) operates on the other buffer. This
increases the responsiveness of camera-using code.

On the Kaluga it makes the following improvements:

  • 320x240 viewfinder at 30fps instead of 15fps using directio
  • 240x240 animated gif capture at 10fps instead of 7.5fps

As discussed at length on Discord, the "usual end user" code will look like
this:

camera = ...

with camera.continuous_capture(buffer1, buffer2) as capture:
    for frame in capture:
        # Do something with frame

However, rather than presenting a context manager, the core code consists of
three new functions to start & stop continuous capture, and to get the next
frame. The reason is twofold. First, it's simply easier to implement the
context manager object in pure Python. Second, for more advanced usage, the
context manager may be too limiting, and it's easier to iterate on the right
design in Python code. In particular, I noticed that adapting the
JPEG-capturing programs to use continuous capture mode needed a change in
program structure.

The camera app was structured as

while True:
    if shutter button was just pressed:
        capture a jpeg frame
    else:
        update the viewfinder

However, "capture a jpeg frame" needs to (A) switch the camera settings and (B)
capture into a different, larger buffer then (C) return to the earlier
settings. This can't be done during continuous capture mode. So just
restructuring it as follows isn't going to work:

with camera.continuous_capture(buffer1, buffer2) as capture:
    for frame in capture:
        if shutter button was just pressed:
            capture a jpeg frame, without disturbing continuous capture mode
        else:
            update the viewfinder

The continuous mode is only implemented in the espressif port; others
will throw an exception if the associated methods are invoked. It's not
impossible to implement there, just not a priority, since these micros don't
have enough RAM for two framebuffer copies at any resonable sizes.

The capture code, including single-shot capture, now take mp_obj_t in the
common-hal layer, instead of a buffer & length. This was done for the
continuous capture mode because it has to identify & return to the user the
proper Python object representing the original buffer. In the Espressif port,
it was convenient to implement single capture in terms of a multi-capture,
which is why I changed the singleshot routine's signature too.

By having a pair of buffers, the capture hardware can fill one buffer while
Python code (including displayio, etc) operates on the other buffer.  This
increases the responsiveness of camera-using code.

On the Kaluga it makes the following improvements:
 * 320x240 viewfinder at 30fps instead of 15fps using directio
 * 240x240 animated gif capture at 10fps instead of 7.5fps

As discussed at length on Discord, the "usual end user" code will look like
this:

    camera = ...

    with camera.continuous_capture(buffer1, buffer2) as capture:
        for frame in capture:
            # Do something with frame

However, rather than presenting a context manager, the core code consists of
three new functions to start & stop continuous capture, and to get the next
frame.  The reason is twofold.  First, it's simply easier to implement the
context manager object in pure Python.  Second, for more advanced usage, the
context manager may be too limiting, and it's easier to iterate on the right
design in Python code.  In particular, I noticed that adapting the
JPEG-capturing programs to use continuous capture mode needed a change in
program structure.

The camera app was structured as
```python
while True:
    if shutter button was just pressed:
        capture a jpeg frame
    else:
        update the viewfinder
```

However, "capture a jpeg frame" needs to (A) switch the camera settings and (B)
capture into a different, larger buffer then (C) return to the earlier
settings. This can't be done during continuous capture mode. So just
restructuring it as follows isn't going to work:

```python
with camera.continuous_capture(buffer1, buffer2) as capture:
    for frame in capture:
        if shutter button was just pressed:
            capture a jpeg frame, without disturbing continuous capture mode
        else:
            update the viewfinder
```

The continuous mode is only implemented in the espressif port; others
will throw an exception if the associated methods are invoked.  It's not
impossible to implement there, just not a priority, since these micros don't
have enough RAM for two framebuffer copies at any resonable sizes.

The capture code, including single-shot capture, now take mp_obj_t in the
common-hal layer, instead of a buffer & length.  This was done for the
continuous capture mode because it has to identify & return to the user the
proper Python object representing the original buffer.  In the Espressif port,
it was convenient to implement single capture in terms of a multi-capture,
which is why I changed the singleshot routine's signature too.
@jepler jepler force-pushed the continuous-capture branch from 554e323 to a8614a6 Compare November 3, 2021 16:02
tannewt
tannewt previously approved these changes Nov 3, 2021
Copy link
Member

@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor suggestion. Up to you if you want to add more.



//| def continuous_capture_stop(self) -> None:
//| """Stop continuous capture"""
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably mention what happens to the buffers as well. I'd assume you'd delete their reference so they may be garbage collected after this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, I added text to cover this. Needs a fresh review now!

Copy link
Member

@tannewt tannewt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@tannewt tannewt merged commit a084124 into adafruit:main Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants