pub enum StreamProcessorRequest {
Show 18 variants
EnableOnStreamFailed {
control_handle: StreamProcessorControlHandle,
},
SetInputBufferPartialSettings {
input_settings: StreamBufferPartialSettings,
control_handle: StreamProcessorControlHandle,
},
SetOutputBufferPartialSettings {
output_settings: StreamBufferPartialSettings,
control_handle: StreamProcessorControlHandle,
},
CompleteOutputBufferPartialSettings {
buffer_lifetime_ordinal: u64,
control_handle: StreamProcessorControlHandle,
},
FlushEndOfStreamAndCloseStream {
stream_lifetime_ordinal: u64,
control_handle: StreamProcessorControlHandle,
},
CloseCurrentStream {
stream_lifetime_ordinal: u64,
release_input_buffers: bool,
release_output_buffers: bool,
control_handle: StreamProcessorControlHandle,
},
Sync {
responder: StreamProcessorSyncResponder,
},
RecycleOutputPacket {
available_output_packet: PacketHeader,
control_handle: StreamProcessorControlHandle,
},
QueueInputFormatDetails {
stream_lifetime_ordinal: u64,
format_details: FormatDetails,
control_handle: StreamProcessorControlHandle,
},
QueueInputPacket {
packet: Packet,
control_handle: StreamProcessorControlHandle,
},
QueueInputEndOfStream {
stream_lifetime_ordinal: u64,
control_handle: StreamProcessorControlHandle,
},
ParticipateInBufferAllocation {
payload: StreamProcessorParticipateInBufferAllocationRequest,
control_handle: StreamProcessorControlHandle,
},
AddBuffer {
payload: StreamProcessorAddBufferRequest,
control_handle: StreamProcessorControlHandle,
},
RemoveBuffer {
payload: StreamProcessorRemoveBufferRequest,
responder: StreamProcessorRemoveBufferResponder,
},
EnableOldOutputBuffers {
control_handle: StreamProcessorControlHandle,
},
EnableSameOutputBufferConcurrentlyInFlight {
control_handle: StreamProcessorControlHandle,
},
EnableForceOutputBuffersFixedImageSize {
control_handle: StreamProcessorControlHandle,
},
#[non_exhaustive] _UnknownMethod {
ordinal: u64,
control_handle: StreamProcessorControlHandle,
method_type: MethodType,
},
}Expand description
Overview of operation:
- Create
- create via CodecFactory - see CodecFactory
- create via LicenseSession - see LicenseSession
- Get input constraints
- OnInputConstraints() - sent unsolicited by stream processor shortly after stream processor creation.
- Provide input buffers ()
- SetInputBufferPartialSettings()
- or AddBuffer, if StreamProcessor reported support for dynamic buffers via CodecFactory - in this case, QueueInputFormatDetails and QueueInputEndOfStream do not require a prior AddBuffer, but QueueInputPacket still does.
- Deliver input data
- QueueInputPacket() + OnFreeInputPacket(), for as long as it takes, possibly working through all input packets repeatedly before…
- Get output constraints and format
- OnOutputConstraints()
- This is not sent until after at least one QueueInput* message is sent by the client, even if the underlying processor behind the StreamProcessor doesn’t fundamentally need any input data to determine its output constraints. This server behavior prevents clients taking an incorrect dependency on the output constraints showing up before input is delivered.
- A client must tolerate this arriving as late as after substantial input data has been delivered, including lots of input packet recycling via OnFreeInputPacket().
- This message can arrive more than once before the first output data.
- Provide output buffers
- SetOutputBufferPartialSettings() / CompleteOutputBufferPartialSettings()
- Data flows, with optional EndOfStream
- OnOutputPacket() / RecycleOutputPacket() / QueueInputPacket() / OnFreeInputPacket() / QueueInputEndOfStream() / OnOutputEndOfStream()
Semi-trusted StreamProcessor server - SW decoders run in an isolate (with very few capabilities) just in case the decoding SW has a vulnerability which could be used to take over the StreamProcessor server. Clients of the stream processor interface using decoders and processing streams of separate security contexts, to a greater extent than some other interfaces, need to protect themselves against invalid server behavior, such as double-free of a packet_index and any other invalid server behavior. Having fed in compressed data of one security context, don’t place too much trust in a single StreamProcessor instance to not mix data among any buffers that StreamProcessor server has ever been told about. Instead, create separate StreamProcessor instances for use by security-separate client-side contexts. While the picture for HW-based decoders looks somewhat different and is out of scope of this paragraph, the client should always use separate StreamProcessor instances for security-separate client-side contexts.
Descriptions of actions taken by methods of this protocol and the states of things are given as if the methods are synchronously executed by the stream processor server, but in reality, as is typical of FIDL interfaces, the message processing is async. The states described are to be read as the state from the client’s point of view unless otherwise stated. Events coming back from the server are of course delivered async, and a client that processes more than one stream per StreamProcessor instance needs to care whether a given event is from the current stream vs. some older soon-to-be-gone stream.
The Sync() method’s main purpose is to enable the client to robustly prevent having both old and new buffers allocated in the system at the same time, since media buffers can be significantly large, depending. The Sync() method achieves this by only delivering it’s response when all previous calls to the StreamProcessor protocol have actually taken effect in the StreamControl ordering domain. Sync() can also be used to wait for the stream processor server to catch up if there’s a possibility that a client might otherwise get too far ahead of the StreamProcessor server, by for example requesting creation of a large number of streams in a row. It can also be used during debugging to ensure that a stream processor server hasn’t gotten stuck. Calling Sync() is entirely optional and never required for correctness - only potentially required to de-overlap resource usage.
It’s possible to re-use a StreamProcessor instance for another stream, and doing so can sometimes skip over re-allocation of buffers. This can be a useful thing to do for cases like seeking to a new location - at the StreamProcessor interface that can look like switching to a new stream.
Variants§
EnableOnStreamFailed
Permit the server to use OnStreamFailed() instead of the server just closing the whole StreamProcessor channel on stream failure.
If the server hasn’t seen this message by the time a stream fails, the server will close the StreamProcessor channel instead of sending OnStreamFailed().
Fields
control_handle: StreamProcessorControlHandleSetInputBufferPartialSettings
When the client is not using dynamic buffers, a single SetInputBufferPartialSettings() provides the StreamProcessor with the client-specified input settings and a BufferCollectionToken which the StreamProcessor will use to convey constraints to sysmem. Both the client and the StreamProcessor will be informed of the allocated buffers directly by sysmem via their BufferCollection channel (not via the StreamProcessor channel).
The client must not QueueInput…() until after sysmem informs the client that buffer allocation has completed and was successful.
The server should be prepared to see QueueInput…() before the server has necessarily heard from sysmem that the buffers are allocated - the server must tolerate either ordering, as the QueueInput…() and notification of sysmem allocation completion arrive on different channels, so the client having heard that allocation is complete doesn’t mean the server knows that allocation is complete yet. However, the server can expect that allocation is in fact complete and can expect to get the allocation information from sysmem immediately upon requesting the information from sysmem.
Mixing of AddBuffer and SetInputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to “now” (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially within the same StreamProcessor.
SetOutputBufferPartialSettings
This is the replacement for SetOutputBufferSettings().
When the client is using sysmem to allocate buffers, this message is used instead of SetOutputBufferSettings()+AddOutputBuffer(). Instead, a single SetOutputBufferPartialSettings() provides the StreamProcessor with the client-specified output settings and a BufferCollectionToken which the StreamProcessor will use to convey constraints to sysmem. Both the client and the StreamProcessor will be informed of the allocated buffers directly by sysmem via their BufferCollection channel (not via the StreamProcessor channel).
Configuring output buffers is required after OnOutputConstraints() is received by the client with buffer_constraints_action_required true and stream_lifetime_ordinal equal to the client’s current stream_lifetime_ordinal (even if there is an active stream), and is permitted any time there is no current stream.
Closing the current stream occurs on the StreamControl ordering domain, so after a CloseCurrentStream() or FlushEndOfStreamAndCloseStream(), a subsequent Sync() completion must be received by the client before the client knows that there’s no longer a current stream.
Mixing of AddBuffer and SetOutputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to “now” (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially not within the same StreamProcessor instance.
See also CompleteOutputBufferPartialSettings().
CompleteOutputBufferPartialSettings
After SetOutputBufferPartialSettings(), the server won’t send OnOutputConstraints(), OnOutputFormat(), OnOutputPacket(), or OnOutputEndOfStream() until after the client sends CompleteOutputBufferPartialSettings().
This message isn’t permitted after AddBuffer. When using dynamic buffers the server can send OnOutputConstraints, OnOutputFormat, OnOutputPacket, or OnOutputEndOfStream at any time after the first QueueInputPacket. In the case of OnOutputPacket there must also be at least one output buffer that’s been added but not yet fully removed (RemoveBuffer not yet complete).
Some clients may be able to send CompleteOutputBufferPartialSettings() immediately after SetOutputBufferPartialSettings() - in that case the client needs to be prepared to receive output without knowing the buffer count or packet count yet - such clients may internally delay processing the received output until the client has heard from sysmem (which is when the client will learn the buffer count and packet count).
Other clients may first wait for sysmem to allocate, prepare to receive output, and then send CompleteOutputBufferPartialSettings().
FlushEndOfStreamAndCloseStream
This message is optional; a client isn’t required to send this ever.
Terminology note: In the name of this message, “flush” means flush through, sometimes called “drain”. This messge does not discard. To discard, just QueueInputPacket with a new stream_lifetime_ordinal without first waiting for OnOutputEndOfStream of the old stream and without sending FlushEndOfStreamAndCloseStream.
There is currently no way to cancel the flush effect of this message short of the client closing the StreamProcessor channel.
This message is only valid after QueueInputEndOfStream() for this stream. The stream_lifetime_ordinal input parameter must match the stream_lifetime_ordinal of the QueueInputEndOfStream(), else the server will close the channel.
A client can use this message to flush through (drain, not discard) the last input data of a stream so that the stream processor server generates corresponding output data for all the input data before the server moves on to the next stream, without forcing the client to wait for OnOutputEndOfStream() before queueing data of another stream.
The difference between QueueInputEndOfStream() and FlushEndOfStreamAndCloseStream(): QueueInputEndOfStream() is a promise from the client that there will not be any more input data for the stream (and this info is needed by some stream processors for the stream processor to ever emit the very last output data). The QueueInputEndOfStream() having been sent doesn’t prevent the client from later completely discarding the rest of the current stream by closing the current stream (with or without a stream switch). In contrast, FlushEndOfStreamAndCloseStream() is a request from the client that all the previously-queued input data be processed including the logical “EndOfStream” showing up as OnOutputEndOfStream() (in success case) before moving on to any newer stream - this essentially changes the close-stream handling from discard to flush-through for this stream only.
A client using this message can start providing input data for a new stream without that causing discard of old stream data. That’s the purpose of this message - to allow a client to flush through (not discard) the old stream’s last data (instead of the default when closing or switching streams which is discard).
Because the old stream is not done processing yet and the old stream’s data is not being discarded, the client must be prepared to continue to process OnOutputConstraints() messages until the stream_lifetime_ordinal is done. The client will know the stream_lifetime_ordinal is done when OnOutputEndOfStream(), OnStreamFailed(), or the StreamProcessor channel closes.
CloseCurrentStream
This “closes” the current stream, leaving no current stream. In addition, this message can optionally release (and unregister) input buffers or output buffers.
If there has never been any active stream, the stream_lifetime_ordinal must be zero or the server will close the channel. If there has been an active stream, the stream_lifetime_ordinal must be the most recent active stream whether that stream is still active or not. Else the server will close the channel.
Multiple of this message without any new active stream in between is not to be considered an error, which allows a client to use this message to close the current stream to stop wasting processing power on a stream the user no longer cares about, then later decide that buffers should be released and send this message again with release_input_buffers and/or release_output_buffers true to get the buffers released, if the client is interested in trying to avoid overlap in resource usage between old buffers and new buffers (not all clients are).
See also Sync().
Fields
control_handle: StreamProcessorControlHandleSync
On completion, all previous StreamProcessor calls have done what they’re going to do server-side, except for processing of data queued using QueueInputPacket().
The main purpose of this call is to enable the client to wait until CloseCurrentStream() with release_input_buffers and/or release_output_buffers set to true to take effect, before the client allocates new buffers and re-sets-up input and/or output buffers. This de-overlapping of resource usage can be worthwhile for media buffers which can consume resource types whose overall pools aren’t necessarily vast in comparison to resources consumed. Especially if a client is reconfiguring buffers multiple times.
Note that Sync() prior to allocating new media buffers is not alone sufficient to achieve non-overlap of media buffer resource usage system wide, but it can be a useful part of achieving that.
The Sync() transits the Output ordering domain and the StreamControl ordering domain, but not the InputData ordering domain.
This request can be used to avoid hitting kMaxInFlightStreams which is presently 10. A client that stays <= 8 in-flight streams will comfortably stay under the limit of 10. While the protocol permits repeated SetInputBufferSettings() and the like, a client that spams the channel can expect that the channel will just close if the server or the channel itself gets too far behind.
Fields
responder: StreamProcessorSyncResponderRecycleOutputPacket
After the client is done with an output packet, the client needs to tell the stream processor that the output packet can be re-used for more output, using this message.
It’s not permitted to recycle an output packet that’s already free with the stream processor server.
If a client is using EnableOldOutputBuffers, the client must recycle all packets when done with them, even those of old buffer_lifetime_ordinal. This is also permitted behavior for all clients.
If a client is not using EnableOldOutputBuffers, the client may optionally omit this message for packets with an old buffer_lifetime_ordinal. In other words, packets from before an explicit or implicit output buffer de-configuration don’t need to be recycled if the client isn’t using and won’t be using EnableOldOutputBuffers.
QueueInputFormatDetails
If the input format details are still the same as specified during StreamProcessor creation, this message is unnecessary and does not need to be sent.
If the stream doesn’t exist yet, this message creates the stream.
The server won’t send OnOutputConstraints() until after the client has sent at least one QueueInput* message.
All servers must permit QueueInputFormatDetails() at the start of a stream without failing, as long as the new format is supported by the StreamProcessor instance. Technically this allows for a server to only support the exact input format set during StreamProcessor creation, and that is by design. A client that tries to switch formats and gets a StreamProcessor channel failure should try again one more time with a fresh StreamProcessor instance created with CodecFactory using the new input format during creation, before giving up.
These format details override the format details specified during stream processor creation for this stream only. The next stream will default back to the format details set during stream processor creation.
This message is permitted at the start of the first stream (just like at the start of any stream). The format specified need not match what was specified during stream processor creation, but if it doesn’t match, the StreamProcessor channel might close as described above.
QueueInputPacket
This message queues input data to the stream processor for processing.
If the stream doesn’t exist yet, this message creates the new stream.
The server won’t send OnOutputConstraints() until after the client has sent at least one QueueInput* message.
When using dynamic buffers the server can send OnOutputConstraints, OnOutputFormat, OnOutputPacket, or OnOutputEndOfStream at any time after the first QueueInputPacket. In the case of OnOutputPacket there must also be at least one output buffer that’s been added but not yet fully removed (RemoveBuffer not yet complete).
The client must continue to deliver input data via this message even if the stream processor has not yet generated the first OnOutputConstraints, and even if the StreamProcessor is generating OnFreeInputPacket for previously-queued input packets. The input data must continue as long as there are free packets to be assured that the server will ever generate the first OnOutputConstraints.
The server will close the channel if this packet refers to an old buffer_lifetime_ordinal. Clients that need to deliver input images of different dimensions to a video encoder can either (a) allocate buffers large enough to contain the range of needed image sizes and use those to deliver all the input images, (b) keep their own sets of buffers used for different image dimensions and re-add old buffers with AddBuffer, moving to a new buffer_lifetime_ordinal each time image dimensions change, or (c) allocate new buffers each time image dimensions change. The (a) option is preferred, for clients that have the flexibiilty to store images of varying dimensions in a single set of buffers. Clients that require each buffer to have a single image size can use option (b) or (c).
QueueInputEndOfStream
Inform the server that all QueueInputPacket() messages for this stream have been sent.
If the stream isn’t closed first (by the client, or by OnStreamFailed(), or StreamProcessor channel closing), there will later be a corresponding OnOutputEndOfStream().
The corresponding OnOutputEndOfStream() message will be generated only if the server finishes processing the stream before the server sees the client close the stream (such as by starting a new stream). A way to force the server to finish the stream before closing is to use FlushEndOfStreamAndCloseStream() after QueueInputEndOfStream() before any new stream. Another way to force the server to finish the stream before closing is to wait for the OnOutputEndOfStream() before taking any action that closes the stream.
In addition to serving as an “EndOfStream” marker to make it obvious client-side when all input data has been processed, if a client never sends QueueInputEndOfStream, no amount of waiting will necessarily result in all input data getting processed through to the output. Some stream processors have some internally-delayed data which only gets drained (pushed through) by additional input data or by this EndOfStream marker. In that sense, this message can be viewed as a drain at InputData domain level, but the drain only takes effect if the stream processor even gets that far before the stream is just closed at StreamControl domain level. This message is not alone sufficient to act as an overall drain at StreamControl level. For that, send this message first and then send FlushEndOfStreamAndCloseStream (at which point it becomes possible to queue input data for a new stream without causing discard of this older stream’s data). Alternately, the client can wait for the OnOutputEndOfStream before closing the current stream.
After a client sends QueueInputEndOfStream for a stream, if the client then sends for the same stream any of QueueInputPacket, QueueInputFormatDetails, QueueInputEndOfStream, the server will close the StreamProcessor channel.
ParticipateInBufferAllocation
This message results in channel closure unless supports_dynamic_buffers is set to true.
This participates in allocation of buffers to be used with AddBuffer later. The client can get VMO handles for these buffers by also participating in the sysmem allocation, using the client’s own related sysmem token (associated with the same logical buffer collection). It’s up to the client to separately set any constraints needed by the client using the client’s own related sysmem token, if any.
Some clients may prefer to use SetInputBufferPartialSettings and/or SetOutputBufferPartialSettings. Servers must support those messages.
In handling this message, if allow_single_buffer is set to true, the
server must not constrain the number of buffers allocated. The server
must set min_buffer_count to 1, and must leave max_buffer_count un-set
or set it to 0xFFFFFFFF, and must leave all min_buffer_count_* fields
un-set. The sender can set min_buffer_count and max_buffer_count to the
same value if the intent is to allocate exactly that many buffers. If
allow_single_buffer is un-set or set to false, the server will
indicate needed buffer counts to sysmem.
The server’s BufferCollection channel (created from the passed-in sysmem2_token) may see ZX_CHANNEL_PEER_CLOSED at any time, but in particular, the server shouldn’t expect the BufferCollection channel to remain connected to sysmem beyond the server sending SetConstraints. For this reason, the server may not be able to call WaitForAllBuffersAllocated or similar, so the server should just send SetConstraints, Close, then close the server’s BufferCollection client_end. This means the server in general shouldn’t attempt to get VMO handles for these buffers while processing this message.
The server should not assume that these buffers will necessarily ever be added with AddBuffer to this StreamProcessor instance or any other StreamProcessor instance (owned by the server or not). These buffers may instead be dropped, or as a less-common example, possibly added to a different codec served by a different server implementation which also participated in the same sysmem buffer collection allocation.
For input buffers, AddBuffer of the allocated buffer(s) to a different StreamProcessor instance of the same codec (same per CodecFactory) is likely to work, but using the same StreamProcessor instance is recommended when feasible.
In contrast, for output buffers, AddBuffer of the allocated buffer(s) to a different StreamProcessor instance of the same codec (same per CodecFactory) can’t (within reason) be made work in general, especially for video decoders. Therefore, for output buffers, the same StreamProcessor instance must be used for this message and AddBuffer. While a client may currently be able to get away with using different StreamProcessor instances for this message and AddBuffer for output buffers for some codecs, this may break at any time without it being considered a server-side bug.
The allocated buffers can later be added using AddBuffer (piecemeal), and can be removed (piecemeal) using RemoveBuffer.
Multiple different ParticipateInBufferAllocation messages can have their
buffers later added to the same StreamProcessor instance using the same
buffer_lifetime_ordinal. This can be useful if the client wants to
allocate buffers incrementally, or dynamically adjust the number of
buffers, potentially while actively processing. See also the
buffer_lifetime_ordinal field of this message.
Server implementations may use sysmem to help verify buffer compatibility later when buffers are added with AddBuffer.
Fields
control_handle: StreamProcessorControlHandleAddBuffer
Add buffers previously created with the help of ParticipateInBufferAllocation.
For input buffers, the client can send QueueInputFormatDetails or QueueInputEndOfStream before any AddBuffer messages. At least one input buffer must be added before a valid QueueInputPacket can be sent. Most clients will want to continue quickly adding buffers up to at least buffer_count_for_server_current to avoid the codec potentially stalling, and typically a low number of buffers beyond that to keep the pipeline running smoothly. Input buffers beyond the first input buffer can be added after the first QueueInputPacket.
Even if buffer_constraints_version_ordinal is current, the server must not close the channel if the buffer isn’t consistent with the current buffer_constraints_version_ordinal (per sysmem GetVmoInfo given consistent StreamProcessor constraints). Instead, the server must send a new OnOutputConstraints. This simplifies some edge cases for some clients, particularly when a client can’t reliably detect whether a newly-obtained buffer was actually allocated after buffer_constraints_version_ordinal changed, or may have been cached from before.
The client can add additional buffers to the same port and buffer_lifetime_ordinal at any time using this message. If the buffer_lifetime_ordinal is no longer the most recent, the message will be ignored, the handle to the buffer dropped, and any later RemoveBuffer message re. the same buffer will complete immediately.
If dynamic_buffers_input_max or dynamic_buffers_output_max is exceeded by the sum of buffers added by all AddBuffer calls with the same buffer_lifetime_ordinal, the server may close the channel. Servers are required to close the channel in this case if performance degradation or un-tested behavior would result from adding too many buffers.
Switching to a new buffer_lifetime_ordinal starts the process of removing buffers associated with an old buffer_lifetime_ordinal. However, until the remove is complete, those buffers can still be used by the codec as normal. See also RemoveBuffer, which can be used to detect when removal is complete, regardless of whether the RemoveBuffer started the removal (including when not using dynamic buffers).
The buffer stays added until removal later completes. The buffer remains added across potentially multiple buffer re-uses. Removal can be initiated (and/or confirmed/fenced) by the client using RemoveBuffer. The codec server can unilaterally initiate buffer removal; if the server does this, it must send a new buffer_constraints_version_ordinal with action_required true. Some other client-initiated messages can also begin buffer removal, such as CloseCurrentStream with release_input_buffers and/or release_output_buffers. The buffers are also automatically removed and released if the client closes the StreamProcessor client_end or the server closes the StreamProcessor server_end.
Mixing of AddBuffer and SetInputBufferPartialSettings / SetOutputBufferPartialSettings is not allowed while there are any buffers still active on the port. To successfully switch modes for a port, most clients will want to just start over with a new StreamProcessor instance. This can also be done reliably by first ensuring that every added buffer under the port up to “now” (including those with older buffer_lifetime_ordinal) has completed a RemoveBuffer request. Only then is it known ok to switch modes for that port under the same StreamProcessor. Most clients will just use one way or the other to add buffers and never need to switch to the other way, especially within the same StreamProcessor.
If a client might plausibly “spam” creation of many new buffer_lifetime_ordinal values without the buffers seeing any actual usage in between, the client should consider starting a Sync every few buffer_lifetime_ordinal(s) to fence cleanup of old buffer_lifetime_ordinal values, and avoid getting ahead of Sync completions by more than 16 buffer_lifetime_ordinal values. Else the channel may close from a backlog of new buffer_lifetime_ordinal(s) getting too far ahead of closing out old ones. The threshold of 16 is well below the enforcement threshold. Clients don’t need to Sync if they won’t be spamming new buffer_lifetime_ordinal values, or if added buffers will see at least some actual usage visible to the client before being replaced again.
All buffers of the same port and buffer_lifetime_ordinal must share the
same [fuchsia.sysmem2/SingleBufferSetttings]. The client can ensure
this in various ways. One way is to use ParticipateInBufferAllocation
then AddBuffer for at least the first buffer, before
ParticipateInBufferAllocation for any subsequent buffers. Another way is
to observe a mismatch in SingleBufferSettings before sending AddBuffer
and bump to the next odd buffer_lifetime_ordinal value for the
AddBuffer.
RemoveBuffer
When using dynamic buffers, this call removes a buffer as soon as the buffer can be removed without adversely impacting any ongoing processing or an in-flight output packet referring to the buffer.
If a client wants to remove a buffer with an in-flight output packet referring to the buffer, the client must RecycleOutputPacket for that in-flight output packet before the RemoveBuffer will complete.
When using SetInputBufferPartialSettings / SetOutputBufferPartialSettings, this call doesn’t initiate removal of the buffer. This call will complete when the buffer is done removing due to other reasons, such as a new buffer_lifetime_ordinal starting.
Until RemoveBuffer completes, the codec is still allowed to send OnOutputPacket messages referencing the buffer, and the codec may still have a VMO handle open to the buffer. After RemoveBuffer completes, the codec guarantees that no subsequent output packet will reference the buffer, and that the server holds no VMO handles to the buffer.
The client may need to recycle an output packet before the RemoveBuffer call can complete. Clients should take care to avoid blocking packet recycling while the RemoveBuffer request is in progress, since this would create a potential deadlock.
The server closing any VMO handles to the buffer prior to completing this call is important for the client’s ability to prevent memory usage spikes.
If the client has “paused” processing by not providing any more input, the client will potentially need to send CloseCurrentStream before RemoveBuffer will complete. When using dynamic buffers, setting release_input_buffers or release_output_buffers to true is not necessary; the RemoveBuffer is explicitly telling the codec to release a specific buffer. When not using dynamic buffers, the client will need to set release_input_buffers or release_output_buffers to true, since RemoveBuffer alone doesn’t initiate removal when not using dynamic buffers. A current “paused” stream needs to be stopped because frames can be held as reference frames, and codecs are never reqiured to copy their output data.
For any video decoder output buffers being removed while there’s an active stream (being fed input or not), it’s typically best for the client to assume that removal of a video decoder output buffer may take a very long duration. This is because bitstreams, especially non-standard-compliant bitstreams, but in some cases potentially even standard-compliant bitstreams, can keep a video decoder output buffer in the set of active reference frames (aka DPB) indefinitely. The server is not required to notice that a stream is not conforming to a bitstream standard in this regard.
In the case of h.264 decode, a standard-complient bitstream will limit the reorder delay to no longer than the max DPB occupancy. However, a server is not required to detect or reject non-compliant streams that potentially keep a frame in the DPB for longer.
For HEVC, the situation is similar to h.264 (IIUC).
In the case of VP9, there is nothing in the bitstream spec that limits the reorder delay (IIUC), meaning a frame can potentially stay in VP9’s set of 8 reference frames (aka DPB) indefinitely. The server is not required to detect or mitigate this.
Until this call completes, the server may still be using the buffer. For output buffers, the server can still send OnOutputPacket message(s) that references this buffer, but only up until the RemoveBuffer completion message is sent by the server. The client must continue to RecycleOutputPacket for packets that reference the buffer, until RemoveBuffer completes.
When using dynamic buffers, upon receiving this message, the server will stop selecting the buffer for any new usage (as in, for any usage that moves the buffer from “free” to “not free” within the server). This applies even if the server has no other buffers available for use (aka no other “free” buffers). Any existing usage of the buffer is not ended early by this call alone.
When not using dynamic buffers, upon receiving this message, the server will just remember to complete this call shortly after the buffer has completed removal triggered by some other cause (removal is not triggered/caused by this call).
For video decoders, when using dynamic buffers, typically the client should take care to avoid removing too many output buffers for continued decode to be possible. If this occurs, the decoder will wait until the client adds another output buffer with AddBuffer. This can be a deadlock if the client never sends that AddBuffer. Due to DPB mechanism(s) and frame reordering, sending a single AddBuffer doesn’t necessarily guarantee another OnOutputPacket, since additional output buffers can be needed before the server can send OnOutputPacket.
The client must not call RemoveBuffer on the same buffer more than once, whether overlapping in time or not. The server should enforce this when not enforcing would require tracking additional concurrent requests. The server is not required to enforce this when enforcing would use more server memory.
The server is allowed to complete this request quickly with success for buffer_lifetime_ordinal and buffer_index combinations that were never real buffers, but must close the channel if the buffer_lifetime_ordinal hasn’t been started by the client yet (no removing potential future buffers).
When using dynamic buffers, a buffer_lifetime_ordinal and buffer_index combination may be re-used after completion of RemoveBuffer, but only if the buffer_lifetime_ordinal is the current buffer_lifetime_ordinal. In other words, no adding buffers under an old buffer_lifetime_ordinal.
Assuming a valid historical buffer is specified, successful completion of this call means the buffer has been fully released by the server and won’t be referenced in any subsequent OnOutputPacket.
EnableOldOutputBuffers
This informs the StreamProcessor that the client is prepared to handle output packets that specify a buffer with buffer_lifetime_ordinal older than the most recent buffer_lifetime_ordinal.
If the client doesn’t send this message, the StreamProcessor will omit any such output, even if DetailedCodecDescription.supports_dynamic_buffers is true. For relevant decoders such as VP9 decoders, not sending this message can result in output that isn’t bistream spec compliant, and the output can be visually different than intended by the bitstream.
Such streams are only possible with some bitstream formats (such as VP9), and are rare, but can happen and can be valid per the bitstream spec. For example, this can be specified by a VP9 bitstream using show_existing_frame to output an old-dimensions buffer after having already output a new-dimensions buffer.
Most clients that send this message will also want to use RemoveBuffer to know when it becomes safe to stop tracking an old buffer.
Most of the time this makes no difference as most bitstreams don’t actually emit old buffers, even if the bitstream spec would allow it. Old output buffers are especially rare for RTC streams which typically don’t have any frame reordering in the first place.
In most video streaming scenarios that use dimension switching as part of their bitrate control strategy (among those that I’ve observed), at the StreamProcessor layer the new dimensions are part of a new stream instead of being spliced together as a continuation of the old stream. That said, using a continuation of the old stream is also a completely valid way to implement dimension switching. When a stream switch occurs as part of dimension switching, the decoder state is not retained and there won’t be any old buffer(s) emitted after new buffer(s), since the new stream doesn’t know anything about old buffers filled by the old stream.
Clients which haven’t tested their ability to handle old output buffers should not send this message. Clients decoding bitstreams like VP9 for decoder compliance testing purposes should send this message (and use a VP9 decoder with DetailedCodecDescrption.supports_dynamic_buffers true). Clients which are required to support old output frames and/or fully comply with a relevant bitstream spec should/must send this message, and should test using a test stream that outputs packets referencing an old output buffer.
Sending this message more than once closes the channel. If sent, this message must be sent prior to the client establishing the first output buffer_lifetime_ordinal. This requirement avoids ambiguity re. free/busy status of packets of old buffer_lifetime_ordinal(s), as the server can auto-recycle packets with old buffer_lifetime_ordinal on behalf of the client when this message was not sent by the client.
This message is only permitted when
[fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers]
is true.
Fields
control_handle: StreamProcessorControlHandleEnableSameOutputBufferConcurrentlyInFlight
This informs the StreamProcessor that the client is prepared to handle output packets that specify the same buffer as another packet that’s also concurrently in flight to the client (not yet recycled).
Most bitstream formats don’t do this. In formats that can do this such as VP9, most actual bitstreams don’t do this.
As an example, in VP9, a stream can cause the same output buffer to be used by another emitted output packet/frame by using show_existing_frame on the same VP9 reference/held frame slot more than once without decoding a new frame into that slot in between.
Clients that need to achieve full bitstream spec compliance for such a bitstream format must send this message.
If this message is not in effect, the server will omit any such output, which can result in output that is not compliant to the relevant bitstream spec, and which can be visually different than the stream intended.
Sending this message more than once closes the channel. If sent, this message must be sent prior to the client establishing the first output buffer_lifetime_ordinal.
This message is only permitted when
[fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers]
is true.
Fields
control_handle: StreamProcessorControlHandleEnableForceOutputBuffersFixedImageSize
For video decoders, this forces the output buffers to be reallocated if the image size needs to change. This is wasteful as it forces extra buffer reallocations given typical video bitrate control strategies (applicable to both streaming and RTC) involving shifting the image dimensions up and down repeatedly as a logical video/stream plays, sometimes even if network conditions remain fairly stable. This extra buffer reallocation cost is incurred vs. baseline whether the dimension switching is achieved within a single StreamProcessor stream or by using a new StreamProcessor stream for new dimensions.
A client should not send this message unless the client really must force the output buffers to be reallocated every time the output image size changes.
Sending this message more than once closes the channel. If sent, this message must be sent prior to any SetInputBufferPartialSettings, SetOutputBufferPartialSettings, ParticipateInBufferAllocation, or AddBuffer.
This message is only permitted if this StreamProcessor is a video decoder.
This message is only permitted when
[fuchsia.mediacodec/CodecFactory.DetailedCodecDescription.supports_dynamic_buffers]
is true.
Fields
control_handle: StreamProcessorControlHandle#[non_exhaustive]_UnknownMethod
An interaction was received which does not match any known method.
Fields
This variant is marked as non-exhaustive
control_handle: StreamProcessorControlHandlemethod_type: MethodTypeImplementations§
Source§impl StreamProcessorRequest
impl StreamProcessorRequest
pub fn into_enable_on_stream_failed( self, ) -> Option<StreamProcessorControlHandle>
pub fn into_set_input_buffer_partial_settings( self, ) -> Option<(StreamBufferPartialSettings, StreamProcessorControlHandle)>
pub fn into_set_output_buffer_partial_settings( self, ) -> Option<(StreamBufferPartialSettings, StreamProcessorControlHandle)>
pub fn into_complete_output_buffer_partial_settings( self, ) -> Option<(u64, StreamProcessorControlHandle)>
pub fn into_flush_end_of_stream_and_close_stream( self, ) -> Option<(u64, StreamProcessorControlHandle)>
pub fn into_close_current_stream( self, ) -> Option<(u64, bool, bool, StreamProcessorControlHandle)>
pub fn into_sync(self) -> Option<StreamProcessorSyncResponder>
pub fn into_recycle_output_packet( self, ) -> Option<(PacketHeader, StreamProcessorControlHandle)>
pub fn into_queue_input_format_details( self, ) -> Option<(u64, FormatDetails, StreamProcessorControlHandle)>
pub fn into_queue_input_packet( self, ) -> Option<(Packet, StreamProcessorControlHandle)>
pub fn into_queue_input_end_of_stream( self, ) -> Option<(u64, StreamProcessorControlHandle)>
pub fn into_participate_in_buffer_allocation( self, ) -> Option<(StreamProcessorParticipateInBufferAllocationRequest, StreamProcessorControlHandle)>
pub fn into_add_buffer( self, ) -> Option<(StreamProcessorAddBufferRequest, StreamProcessorControlHandle)>
pub fn into_remove_buffer( self, ) -> Option<(StreamProcessorRemoveBufferRequest, StreamProcessorRemoveBufferResponder)>
pub fn into_enable_old_output_buffers( self, ) -> Option<StreamProcessorControlHandle>
pub fn into_enable_same_output_buffer_concurrently_in_flight( self, ) -> Option<StreamProcessorControlHandle>
pub fn into_enable_force_output_buffers_fixed_image_size( self, ) -> Option<StreamProcessorControlHandle>
Sourcepub fn method_name(&self) -> &'static str
pub fn method_name(&self) -> &'static str
Name of the method defined in FIDL