drivers: video: API enhancement

[josuah]

Introduction

Pursue the work on the video API past the Video 4 Zephyr added to Zephyr in #17194.

Problem description

I would like to implement UVC and enable more complex video pipelines to be written without having to rewrite from scratch a new application every time the devicetree is updated.

Proposed change

Incrementally rework the devicetree and video driver API

Detailed RFC

The existing API leaves corners left for future specification.

The goal is:

• To offer a way to fully access the devicetree video content from the application
• Allow to effectively connect video elements together in code, like specified in the devicetree without having to do it manually.
• Define all semantics to cover the very most common use-cases completely.
• Reuse ideas and code widespread through other Zephyr drivers.

Proposed change (Detailed)

remote-endpoint is not used by Zephyr drivers. RTIO can be leveraged to replace the many FIFOs that are typically defined in each driver 1, 2, 3 and provide a way to directly connect drivers while also allowing the application to be interleaved in the pipeline. This would act as a splice(2) for Zephyr video drivers, triggered by the devicetree `remote-endpoint

• drivers: video: "remote-endpoint" does nothing #72311 (comment)

video_endpoint_id has unclear semantics in some cases (doc fix only) and can be adjusted to allow positive numbers refer an individual endpoint number or address

• drivers: video: introduce semantics for enum endpoint_id #73009

Following this, devicetree macros can be introduced to provide ways to refer a particular endpoint without manually coding the endpoint number from the application.

• drivers: video: Introduce macro to deal with endpoints #73023

video_get_caps() fills a struct with all the video caps filled at once. This works well, but makes it more difficult than using an enumerator-style API such as what #72254 proposes which allows drivers to be a bit more generic: a device doing software video processing can filter the format capabilities of its sources this way.

• Video : Add frame interval APIs #72254

video_enqueue(): "an addition to the video API to enable streaming partial frames within one video buffer" is missing. A new API is introduced as part of Arducam Mega sensor support.

• drivers: video: video_mcux_smartdma: add SMARTDMA video driver #72827
• drivers: video: Add support for arducam mega #66994 (comment)

Introduce a directory for the video drivers, as sensors, MIPI controllers, DMA controllers, software blocks [...] are being mixed in one same driver API. A sensor directory was the chosen way and will be merged once the ongoing PR are completed.

• drivers: video: Move the sensors to their dedicated directory #73013

Add a drivers/video_*_skeleton.c in the same style as drivers/usb/udc/udc_skeleton.c that can speed-up development and help with understanding the semantics.

• drivers: video: sensors: add a sensor_skeleton.c to speed-up contribution of simple sensors #73867

The devicetree can be flattened to enable more uniform processing, such as DT macros used to reduce boilerplate code and homogenize drivers

• dts: flatten video's device tree endpoints definition #72950 (this turns out to not require any change: just conventions)

Dependencies

Annotated inline above.

Concerns and Unresolved Questions

• A few modification to existing drivers are to be introduced to support all proposed changes.
• Many of the "rework" proposal are a matter of deciding what is most comfortable to work with, and a call to discussion more than a fixed solution proposal.

Alternatives

Leaving-up the application developer resolve these issues is the current approach, and demos and samples submitted #71463 show that it effectively works end-to-end with the current API.

For the case of RTIO, it is also possible to introduce a "genderless" interconnect between source and sinks (so that the application or another driver can use it) by replacing enqueue()/dequeue() by a public FIFO integrated directly in the API.

[josuah]

Extra material to investigate the rationale of the existing APIs, so that their semantics can be kept: https://static.linaro.org/connect/san19/presentations/san19-503.pdf

[josuah]

Thanks to @loicpoulain I rediscovered Gstreamer, and it is great!

The above is also the foundation for a zephyr-shell pipeline, as it is not possible to do it if rewriting the application every time is required. The difficulty, though, is that getting <&phandles> from remote-endpoint at compile-time will not allow to reconnect them later with zephyr-shell.

Maybe it is enough to have "gst-launch pipeline, but all devicetree-made" as a first step...

[teburd]

Right you could treat an rtio_iodev as an endpoint (input or output stream), which is what sensors do today. There's effectively two output streams for sensors, one which is polled, and one which is driven by the sensor itself with an event.

The sensing subsystem provides what amounts to a statically defined pipeline with fanout/fanin possibilities and builds on top of this.

I could imagine video input/output endpoints being similar, though perhaps needing some additional information about image encoding and framesize to describe the metadata.

Image sensors (video) are after all sensors :-)

[josuah]

So let's give it a try! :) I will pick some hardware known to work first and move from here.

To everyone working on a sample or a driver, I wish to avoid disturbing any ongoing effort, so in case I missed an issue, do feel free to ping me about it, I can provide a patch to merge on the PR or delay another PR so that actual driver contribution gets merged first!

[josuah]

Some insightful overview of how libcamera and V4L2 APIs came-up to be:
https://archive.fosdem.org/2022/schedule/event/mobile_camera/

Key takeaways (from the slides):

• Application might want to configure itself each point in the pipeline
• A system should not hard-code use-cases
• There will be vendor-specific controls