DesignVideo

Version 17 (Adrian Georgescu, 09/04/2010 02:15 pm)

1 2 Adrian Georgescu
[[TOC(Design*, depth=1)]]
2 2 Adrian Georgescu
3 17 Adrian Georgescu
= Video Blueprint =
4 3 Adrian Georgescu
5 3 Adrian Georgescu
Design and implement Ticket [ticket:18]
6 4 Saúl Ibarra Corretgé
7 4 Saúl Ibarra Corretgé
== Goals ==
8 4 Saúl Ibarra Corretgé
9 17 Adrian Georgescu
The goal is to implement the support for p2p and multi-party video sessions is the SIPSIMPLE middleware.  VP8 and H.264 codecs must be supported.
10 1 Adrian Georgescu
11 11 Adrian Georgescu
The video stream should behave like the audio stream at the transport level: SRTP and ICE should be usable. It must comply with the IMediaStream interface, the same way AudioStream does.
12 11 Adrian Georgescu
13 11 Adrian Georgescu
== Integration ==
14 11 Adrian Georgescu
15 11 Adrian Georgescu
In the picture below the integration of the video support into SIPSIMPLE is shown divided into it's different components:
16 11 Adrian Georgescu
17 14 Adrian Georgescu
[[Image(video_design.png)]]
18 11 Adrian Georgescu
19 11 Adrian Georgescu
 * PJSIP: a new transport is needed. Instead of creating a whole new transport, which would require to implement SRTP and ICE again, a transport adapter will be implemented. A transport adapter sits between a real transport and a pjmedia_stream. To the stream, this adapter will look like a media transport, and to existing media transport, this adapter will look like a stream. The benefit of this approach is we can use the same adapter for both kind of media transports, that is the UDP and ICE media transport. This is exactly the approach that was taken for SRTP. This transport adapter will be responsible for encoding/decoding the video information.
20 11 Adrian Georgescu
21 11 Adrian Georgescu
 * Middleware:
22 12 Adrian Georgescu
23 11 Adrian Georgescu
  * VideoTransport: the VideoTransport will use RTPTransport to carry the video data and will be responsible for building the SDP for the video stream. A new option will be added to RTPTransport so that it starts the video transport adapter instead of the regular transport when needed.
24 12 Adrian Georgescu
25 11 Adrian Georgescu
  * VideoStream: implements IMediaStream interface. Will export a plugable mechanism so that the application layer can access the video data and display it in a window for example, similar to ExternalVNCViewer on MSRP streams.
26 1 Adrian Georgescu
27 11 Adrian Georgescu
 * Application: the application will receive the video data from the stream and 'paint' it on a window.
28 11 Adrian Georgescu
29 17 Adrian Georgescu
== Video Acquisition ==
30 4 Saúl Ibarra Corretgé
31 12 Adrian Georgescu
First approach will be to create pjmedia_videostream object which will do the video acquisition at low level and pass it to transport_video.
32 12 Adrian Georgescu
33 1 Adrian Georgescu
== Roadmap ==
34 1 Adrian Georgescu
35 12 Adrian Georgescu
This milestones should be achieved in order to get video working:
36 7 Saúl Ibarra Corretgé
37 12 Adrian Georgescu
 * SDP negotiation: make SIPSIMPLE able to generate and negotiate a valid video SDP.
38 1 Adrian Georgescu
 * Null video: SIPSIMPLE will generate a valid RTP payload with dummy data that will be exchanged after a successful SDP negotiation.
39 12 Adrian Georgescu
 * Video reception an still image sending: add the ability to receive and display the remote video stream and generate a valid video stream from an still image.
40 12 Adrian Georgescu
 * Video acquisition: send real video data.
41 9 Saúl Ibarra Corretgé
42 17 Adrian Georgescu
== Encoder and Acquisition Choices ==
43 1 Adrian Georgescu
44 1 Adrian Georgescu
 * libVLC can be used for encoding and decoding the video data. It has ctypes based Python bindings, that should be used at the application level to display remote video. Acquisition will most likely be done in C. A libVLC shared library will need to be built with all necessary module statically linked: h264 encoder/decoder, core modules, etc.
45 1 Adrian Georgescu
 * ffmpeg may be used instead of libvlc, is more lightweight
46 1 Adrian Georgescu
47 17 Adrian Georgescu
== Video Mixer ==
48 15 Adrian Georgescu
49 17 Adrian Georgescu
For multi-party conferencing  scenario, the party that is the  mixer must overlay its own video with the inputs from the conference participants RTP video streams into a new composed screen that is then sent back to the participants. 
50 17 Adrian Georgescu
51 17 Adrian Georgescu
http://freej.org/ is a multi-layer video mixer C++ with Python bindings
52 17 Adrian Georgescu
53 17 Adrian Georgescu
The active speaker must be rendered more prominently than the listening parties. The video overlay must be dynamically changed by detecting the party that speaks louder from the RTP audio stream. The active speaker takes the top side of the screen while the other participants must be rendered in thumbnail mode horizontally.