Repository navigation
pipeline: rpi: pisp: Support LSB-aligned unpacked RAW streams - #365
bertouttier wants to merge 1 commit into
Conversation
The PiSP Front End can shift its 16-bit output right, and the Back End can shift its input left by the same amount (the SHIFT field of PISP_FE_OUTPUT0_FORMAT and PISP_BE_INPUT_FORMAT). That writes unpacked 10 to 14-bit samples aligned to the LSB, the layout of the unpacked formats such as formats::SBGGR10 and SBGGR12, which some applications want without the cost of shifting every frame on the CPU. Derive the shift of unpacked Bayer formats with fewer than 16 bits from their bit depth in toPiSPImageFormat(), the way the compressed formats give the compression mode: 16 - bit depth. Keep an unpacked RAW stream request at the sensor bit depth as the CFE device format when the CFE Front End output offers it. The Back End input format comes from the same device format, so the ISP outputs keep working alongside the RAW stream. The CFE needs to offer the unpacked formats on its Front End output (rp1_cfe in the Raspberry Pi kernel). Without that, the request falls back to the unpacked 16-bit format as before. Tested on a Raspberry Pi 5 with a 10-bit and a 12-bit sensor: the RAW stream is the 16-bit one shifted down by 6 and 4 bits, and the ISP output is bit-identical with and without the shift; with an unpatched kernel, the stream falls back to 16 bits. Signed-off-by: Bert Outtier <bert.outtier@spectricity.com>
|
May I ask what's the use case you might be trying to solve here? We've purposely left out the LS aligned formats so that all downstream processing can be kept as generic 16-bit operations regardless of upstream bit-depth. In addition, the kernel driver and userland components for setting up both the FE and BE can be kept simpler with regards to format handling. |
|
Hi, and thanks for taking a look! The use case is a custom RAW processing pipeline that runs on several SoCs, Pi 5 among them. On the others the capture path delivers unpacked RAW10/RAW12 aligned to the LSB, i.e. the standard V4L2/libcamera SBGGR10/SBGGR12 layout. The pipeline is built around that: its calibration data, thresholds and saturation handling all work in native sensor counts (0..1023 / 0..4095). On the Pi we currently have to shift every frame on the CPU, or report SBGGR16 and keep the shift on the side. Since the FE can do it for free, it seemed nicer to let the hardware do it and report the honest format. I do take the point about format handling complexity. We're happy to keep it downstream if it doesn't fit your plans. I just thought it would be a nice addition and I assume the SHIFT field in the PISP_FE_OUTPUT0_FORMAT register of the PiSP was meant for this purpose? |
Ah, ok this makes sense!
Unfortunately I think there's a few more things that will need changing to fully support this in libcamera. In particularly, the IPA tuning algorithms assume 16-bit sample values, so things like black level will need updating per-bitdepth. For now, I would suggest keep this downstream only if that's ok? I will consider all the implications and perhaps address this in a future change. |
The kernel change is in rp1_cfe rather than rp1-cfe, so that is the downstream only kernel driver for the CFE. We'll have the joy of switching to the upstream driver at some point once streams support actually works. |
The PiSP Front End can shift its 16-bit output right, and the Back End can shift its input left by the same amount: the
SHIFTfield ofPISP_FE_OUTPUT0_FORMATandPISP_BE_INPUT_FORMAT, which libpisp exposes asPISP_IMAGE_FORMAT_SHIFT_n. With that, the CFE can write unpacked 10 to 14-bit samples aligned to the LSB: the layout offormats::SBGGR10,SBGGR12and the other unpacked Bayer formats. Some applications want RAW in exactly that layout, without shifting every frame on the CPU.Today an unpacked RAW request below 16 bits is always adjusted to the 16-bit (MSB-aligned) format.
This PR gives unpacked Bayer formats the same treatment the compressed formats already get, where the pixel format selects the hardware mode:
toPiSPImageFormat()setsPISP_IMAGE_FORMAT_SHIFT_(16 - bit depth)for unpacked Bayer formats with fewer than 16 bits:platformValidate()keeps an unpacked RAW request at the sensor bit depth (10 to 14 bits) as the CFE device format, if the CFE Front End output node offers that format. Otherwise it falls back to the 16-bit format as before.The ISP outputs keep working alongside the LSB-aligned RAW stream, since the BE input shifts the samples back.
This needs the CFE to offer the unpacked formats on its FE output: raspberrypi/linux#7722. With a kernel without it, the CFE node doesn't list the formats, and the behaviour is unchanged.
Testing
On a Raspberry Pi 5 with a 10-bit and a 12-bit CSI-2 Bayer sensor, with the sensor test pattern. This change was tested on top of the sensors' support patches, since neither sensor is supported upstream. The tested version differs from this commit only by a local variable.