Use AVCaptureVideoDataOutput on capture.py

[0dm]

WIP to resolve #570

[abrichr]

Via ChatGPT:

Given your implementation and the issue you're facing with saving images, a few areas could potentially cause the delegate method not to trigger as expected, thereby not saving the images. Let's review and suggest corrections:

1. Ensure Delegate Persists

One common issue with delegates in Objective-C (and by extension, when using PyObjC) is the delegate being deallocated if it's not strongly referenced. Since you're using a concurrent futures thread pool executor as the queue for the delegate callbacks, ensure that the delegate instance (VideoDelegate) persists for the duration of the capture session. If it's deallocated, your callback won't be called.

In your Capture class, you should keep a strong reference to the VideoDelegate instance:

self.delegate = VideoDelegate.alloc().init()

Then use self.delegate when setting the sample buffer delegate:

self.video_data_output.setSampleBufferDelegate_queue_(self.delegate, queue)

2. Confirming Delegate Method Signature

Ensure the method signature in your VideoDelegate matches what AVFoundation expects. The method signature looks correct based on your code, but it's always a good idea to double-check this against the AVFoundation documentation and ensure no recent changes might affect it.

3. Debugging Delegate Calls

To verify if the issue lies with the delegate not being called, add more logging inside your delegate method and at key points in your Capture class setup to ensure each step is executed as expected. For example:

from Foundation import NSLog

class VideoDelegate(NSObject):
    def captureOutput_didOutputSampleBuffer_fromConnection_(self, output, sampleBuffer, connection):
        NSLog("Delegate method called")
        # Your existing code...

4. Thread Pool Executor

Your use of concurrent.futures.ThreadPoolExecutor as the queue for delegate callbacks is unconventional. Typically, an AVCaptureVideoDataOutput expects a dispatch queue (created with dispatch_queue_create) for its second argument in setSampleBufferDelegate_queue_. Mixing Python's thread pool executor with Objective-C's expectation might be leading to unexpected behavior.

Change your queue to use a dispatch queue:

import objc
self.queue = objc.dispatch_queue_create("videoDataOutputQueue", None)
self.video_data_output.setSampleBufferDelegate_queue_(self.delegate, self.queue)

5. Verify Configuration and Permissions

• Ensure your application has the necessary permissions to capture the screen and use the camera or microphone if enabled.
• Verify that your session configuration (adding inputs and outputs) completes successfully. If there's an issue with how the session is configured, it may not start correctly, resulting in no callbacks to the delegate.

Summary

Adjust your code to maintain a strong reference to the delegate and use a dispatch queue for the delegate callbacks. Ensure your app has the necessary permissions and add more extensive logging to help pinpoint where the issue might be occurring. If these steps don't resolve the problem, further investigation into the session's configuration and state when trying to start it would be warranted.

[abrichr]

Excellent work @0dm !

I believe the next step is to modify utils.take_screenshot to return the latest version of this data.

What do you think about accepting a write queue as a parameter to VideoDelegate? Then that object could optionally be passed into utils.take_screenshot.

[abrichr]

Why remove audio and camera functionality?

[0dm]

There isn't much of a use for those because we aren't capturing movies anymore. I can just duplicate the old file to keep that functionality separate (call it like movie.py) or something.

[0dm]

I added it back as a separate class, MovieCapture
e1682ba

[abrichr]

🎉 🚀

[abrichr]

@0dm can you please remind me what is the correct way to run this?

(openadapt-py3.10) abrichr@MacBook-Pro-4 OpenAdapt % python openadapt/capture/_macos.py  
Press enter to stopobjc[7309]: class `NSKVONotifying_AVCaptureConnection' not linked into application

[0dm]

@abrichr

I believe the next step is to modify utils.take_screenshot to return the latest version of this data.

What do you think about accepting a write queue as a parameter to VideoDelegate? Then that object could optionally be passed into utils.take_screenshot.

I don't think it's very easy to work directly with the VideoDelegate object because we cannot call it with arguments, we just pass it to video_data_output.setSampleBufferDelegate_queue_. What we could do instead is use a global write queue or maybe a named pipe from Apple's API.

@0dm can you please remind me what is the correct way to run this?

Running it as a python module doesn't seem to work (python -m openadapt.capture._macos)

  File "/Users/aaron/GitHub/OpenAdapt/openadapt/capture/_macos.py", line 42, in <module>
    class VideoDelegate(NSObject):
objc.error: VideoDelegate is overriding existing Objective-C class

So the current method is the one you had, python3 openadapt/capture/_macos.py. There is that warning
objc[7309]: class `NSKVONotifying_AVCaptureConnection' not linked into application, but it doesn't seem to affect the functionality.