fix(core): Improve recording pathway and keyframe behavior

[ehfd]

Fixes quirks in a9f1993.

Caller-visible cost	old (pre-TheLamer)	TheLamer (a9f1993)	now (working tree)
Stalled recorder — max	0.2 ms	106.7 ms	0.013 ms
Stalled recorder — avg	11 µs	100 µs	0 µs
Healthy recording @1MiB frames — avg	578 µs	210 µs	2 µs
Healthy recording @1MiB frames — max	978 µs	600 µs	13 µs
Idle (socket configured, no recorder)	0 µs	0 µs	0 µs
Byte-exact header-stripped stream	✓	✓	✓
IDR on connect	✓	✓	✓
Forced-IDR tax per 300 recorded frames	5	0	0

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[thelamer]

I don't like this from a code maintenance standpoint, the recording sink was introduced out of an external PR here #15. Since its introduction it is something I have never used and I doubt many of our users are. It requires a session to be running and producing frames to even function with the most reliable way to use it by flagging the stream with streaming mode so it is always producing frames. The introduction of the keyframes is to support recording it as a raw .h264 stream and not re-encoding with ffmpeg.

So down the line for v2.0.0+ I do not want it to be another thing that is needed to be regression tested against hevc, av1, etc encoders. We should just mem copy, get some normal keyframes like a recording would expect, and spit out exactly what the browser client is getting from a frame standpoint.

In general the whole system needs a makeover to be something useful, the initial idea is we are getting "free" recording ability leveraging existing tech, but to be truly useful it kind of needs to be it's own pipe. A user that wants to record a session really just wants an mp4 to be spit out in the end and it really should not depend on active clients. This makes it useful for CI scenarios and any other agentic automation, actions are performed in the desktop session and you start and stop the recording while those actions are being performed, then you get a properly encoded video file that can be used out of the box to present the data in a web browser or anywhere else.

As this tech stack exists it is really only useful impo for corporate envs where you are recording your users to audit them.

[ehfd]

@thelamer Will you take it if I reimplement it on a maintainable way?

[thelamer]

I mean do you share my concern? Like do we really need this to be embedded in all the encoders ?

Lets just intercept the the delivery pipeline for now and ask for keyframes when needed. The final implementation of this I see it being able to run independently of selkies completely thus the env var PIXELFLUX_. just in general it would be nice if all setting can auto adapt to using env vars, pythonic stuff , or cli flags like the selkies settings.

Eventually land on where you can make a very short python script to spin up a wayland socket, use the rest api to control it via CU, and record what happens in the session. A more general purpose selenium or pyautogui type thing than a full desktop delivery stack to a web browser. Use this for a battle tested virtual display manager with gpu support to run desktop applications in test envs for x11 and wayland. I think the accepted solution for many of these is just to run xvfb right now.

But I don't want to develop this all now, lets get 2.0.0 cut, lets get you the wheels you need to build out the next iteration of Selkies, and lets earmark this for complete refactor while it still provides the same basic functionality even if degraded it used to.

[ehfd]

I mean do you share my concern? Like do we really need this to be embedded in all the encoders ?

Of course not. I agree.

Let me propose a compromise, then.