[Spice-devel,0/11] Add GStreamer and VP8 support

Submitted by Francois Gouget on May 28, 2015, 2:17 p.m.

Details

Message ID alpine.DEB.2.11.1505271806460.2781@amboise
State New
Headers show

Not browsing as part of any series.

Commit Message

Francois Gouget May 28, 2015, 2:17 p.m.
On Tue, 26 May 2015, Fabio Fantoni wrote:
[...]
> After din't show gstreamer warning anymore but still have image freeze
> and also spice-gtk crash after open video fullscreen, here the full gdb
> datas:
> http://pastebin.com/idTkZLh0

It looks like the crash happened when trying to access the st->out_frame 
buffer that was set up by stream_gst_data() in the client.

I don't know why that would be the case: as far as I can tell that 
pointer is either valid or NULL; even if we return early from 
push_frame() or pull_frame().

It may not be related to the issue you've run into but the patch below 
should help quite a bit if you try running the client more than a minute 
or two:

I also have a patch that avoids copying the out_frame buffer but given 
the low CPU usage of the client that should not be an issue.


> After I tried with gstreamer using ffmpeg, vp8 doesn't crashed, probably
> was problem of gstreamer0.10-plugins-bad but "image freeze" problem remain

When does the freeze happen? A short freeze is normal during the 
transition from the regular transport to the video streaming but that 
happens in the mjpeg case too. Is the freeze temporary?


[...]
> About vp8 image freeze here some seconds of gst log debug on spice-gtk
> when problem happen: http://pastebin.com/PP2R43Yf

Nothing jumped at me in this log.


> Using spice:vp8 seems only have low performance.

In my experience the VP8 encoder saturates a core which is why it is not 
smooth. When running a test pipeline through gst-laucnh I'm able to 
solve that by playing with vp8enc's speed and threads parameters but for 
some reason these have no effect in Spice.

gst-launch videotestsrc ! video/x-raw-rgb,width=1024,height=768 ! \
           ffmpegcolorspace ! vp8enc speed=2 threads=4 ! \
           vp8dec ! ffmpegcolorspace ! fpsdisplaysink

Patch hide | download patch | download mbox

diff --git a/gtk/channel-display-gst.c b/gtk/channel-display-gst.c
index b880ce4..9da078a 100644
--- a/gtk/channel-display-gst.c
+++ b/gtk/channel-display-gst.c
@@ -221,6 +221,7 @@  static void pull_frame(display_stream *st)
 
         // TODO seems like poor memory management
         if (gst_memory_map(memory, &mem_info, GST_MAP_READ)) {
+            g_free(st->out_frame);
             st->out_frame = g_malloc0(mem_info.size);
             memcpy(st->out_frame, mem_info.data, mem_info.size);


Comments

Il 28/05/2015 16:17, Francois Gouget ha scritto:
> On Tue, 26 May 2015, Fabio Fantoni wrote:
> [...]
>> After din't show gstreamer warning anymore but still have image freeze
>> and also spice-gtk crash after open video fullscreen, here the full gdb
>> datas:
>> http://pastebin.com/idTkZLh0
> It looks like the crash happened when trying to access the st->out_frame 
> buffer that was set up by stream_gst_data() in the client.
>
> I don't know why that would be the case: as far as I can tell that 
> pointer is either valid or NULL; even if we return early from 
> push_frame() or pull_frame().
>
> It may not be related to the issue you've run into but the patch below 
> should help quite a bit if you try running the client more than a minute 
> or two:
>
> diff --git a/gtk/channel-display-gst.c b/gtk/channel-display-gst.c
> index b880ce4..9da078a 100644
> --- a/gtk/channel-display-gst.c
> +++ b/gtk/channel-display-gst.c
> @@ -221,6 +221,7 @@ static void pull_frame(display_stream *st)
>  
>          // TODO seems like poor memory management
>          if (gst_memory_map(memory, &mem_info, GST_MAP_READ)) {
> +            g_free(st->out_frame);
>              st->out_frame = g_malloc0(mem_info.size);
>              memcpy(st->out_frame, mem_info.data, mem_info.size);
>
> I also have a patch that avoids copying the out_frame buffer but given 
> the low CPU usage of the client that should not be an issue.
>
>
>> After I tried with gstreamer using ffmpeg, vp8 doesn't crashed, probably
>> was problem of gstreamer0.10-plugins-bad but "image freeze" problem remain
> When does the freeze happen? A short freeze is normal during the 
> transition from the regular transport to the video streaming but that 
> happens in the mjpeg case too. Is the freeze temporary?
>
>
> [...]
>> About vp8 image freeze here some seconds of gst log debug on spice-gtk
>> when problem happen: http://pastebin.com/PP2R43Yf
> Nothing jumped at me in this log.
>
>
>> Using spice:vp8 seems only have low performance.
> In my experience the VP8 encoder saturates a core which is why it is not 
> smooth. When running a test pipeline through gst-laucnh I'm able to 
> solve that by playing with vp8enc's speed and threads parameters but for 
> some reason these have no effect in Spice.
>
> gst-launch videotestsrc ! video/x-raw-rgb,width=1024,height=768 ! \
>            ffmpegcolorspace ! vp8enc speed=2 threads=4 ! \
>            vp8dec ! ffmpegcolorspace ! fpsdisplaysink
>
>
>

Thanks for reply, I'm preparing spice-gtk with vp8 patches armhf for
test it on arm thin client (for example raspberry 2), I want try to use
gst-omx for hardware decoding of mjpeg and vp8, can I simply build and
install gst-omx instead of gst-ffmpeg or a change in spice-gtk is needed
for use gst-omx?

Tomorrow I'll do other tests (including updating spice-server patches)
and I'll try to understand what can be the temp. image freeze and other
important performance problem.

Thanks for any reply and sorry for my bad english.
On Thu, 28 May 2015, Fabio Fantoni wrote:
[...]
> Thanks for reply, I'm preparing spice-gtk with vp8 patches armhf for
> test it on arm thin client (for example raspberry 2), I want try to use
> gst-omx for hardware decoding of mjpeg and vp8, can I simply build and
> install gst-omx instead of gst-ffmpeg or a change in spice-gtk is needed
> for use gst-omx?

I have patches that could probably be useful to you for this: a 
simplification of the pipeline creation, optionally the use of 
decoebin, and for the performance aspects zero-copy for both the input 
buffer and the ouput frame. So I have created some github repositories 
to make it easier to get these:


https://github.com/fgouget/spice-gtk.git
 * gst branch
   All the above except for decodebin.

 * decodebin branch
   Uses decodebin instead of manually picking the decoders. However on 
   my machine decodebin tries to use vaapi which, given my test 
   configuration, immediately triggers an assert, thus killing the 
   process. Uninstalling vaapi fixes that but it's annoying. So I don't 
   really like the decodebin approach though it's probably more a vaapi 
   bug than a decodebin issue.


https://github.com/fgouget/spice.git
 * gst branch
   The patches adding GStreamer and VP8 support.

 * gst-1.0 branch
   Experimental branch with non-functional GStreamer 1.0 and h264 code. 
   Note that GStreamer 1.0 support can be disabled using './configure 
   --disable-gstreamer-1.0' so H264 support can be tested too.


https://github.com/fgouget/xf86-video-qxl.git
 * gst branch
   Has the patch to make it possible to specify the encoder:codec 
   pairs in either SpiceVideoCodecs or XSPICE_VIDEO_CODECS.


There are also spice-common and spice-protocol repositories but these 
should get pulled automatically.
Il 29/05/2015 04:29, Francois Gouget ha scritto:
> On Thu, 28 May 2015, Fabio Fantoni wrote:
> [...]
>> Thanks for reply, I'm preparing spice-gtk with vp8 patches armhf for
>> test it on arm thin client (for example raspberry 2), I want try to use
>> gst-omx for hardware decoding of mjpeg and vp8, can I simply build and
>> install gst-omx instead of gst-ffmpeg or a change in spice-gtk is needed
>> for use gst-omx?
> I have patches that could probably be useful to you for this: a 
> simplification of the pipeline creation, optionally the use of 
> decoebin, and for the performance aspects zero-copy for both the input 
> buffer and the ouput frame. So I have created some github repositories 
> to make it easier to get these:
>
>
> https://github.com/fgouget/spice-gtk.git
>  * gst branch
>    All the above except for decodebin.
>
>  * decodebin branch
>    Uses decodebin instead of manually picking the decoders. However on 
>    my machine decodebin tries to use vaapi which, given my test 
>    configuration, immediately triggers an assert, thus killing the 
>    process. Uninstalling vaapi fixes that but it's annoying. So I don't 
>    really like the decodebin approach though it's probably more a vaapi 
>    bug than a decodebin issue.

Thanks, I'll try it.

>
>
> https://github.com/fgouget/spice.git
>  * gst branch
>    The patches adding GStreamer and VP8 support.
>
>  * gst-1.0 branch
>    Experimental branch with non-functional GStreamer 1.0 and h264 code. 
>    Note that GStreamer 1.0 support can be disabled using './configure 
>    --disable-gstreamer-1.0' so H264 support can be tested too.

I'm trying the gst1 support but build fails with this error:
gstreamer_encoder.c: In function 'construct_pipeline':
gstreamer_encoder.c:219:41: error: 'GstEncoder' has no member named 'frc'
                      "deadline", encoder->frc.period * 1000 / 2,

Why you tell that 1.0 is not functional? I suppose not only for the
build error I had.

>
>
> https://github.com/fgouget/xf86-video-qxl.git
>  * gst branch
>    Has the patch to make it possible to specify the encoder:codec 
>    pairs in either SpiceVideoCodecs or XSPICE_VIDEO_CODECS.
>
>
> There are also spice-common and spice-protocol repositories but these 
> should get pulled automatically.
>
>
On Fri, 29 May 2015, Fabio Fantoni wrote:
[...]
> I'm trying the gst1 support but build fails with this error:
> gstreamer_encoder.c: In function 'construct_pipeline':
> gstreamer_encoder.c:219:41: error: 'GstEncoder' has no member named 'frc'
>                       "deadline", encoder->frc.period * 1000 / 2,
> 
> Why you tell that 1.0 is not functional? I suppose not only for the
> build error I had.

It's a merge bug. Sorry. I have pushed an update that fixes this so you 
may want to rebase.

The initial push had caps issues, mostly it was missing format=BGRx, 
which caused GStreamer 1.0 not to work. Note that the failure mode 
caused the video to be sent through the regular non-stream channel so it 
looked like it worked.

Now the GStreamer 1.0 situation is as follows:
 * VP8 works.
 * MJPEG fails because of a caps negotiation error despite the 
   videoconvert element.
 * H264 seems to work but the client fails to display it. So I'm not 
   totally sure.

Also I finally found the right parameters to improve the VP8 encoder 
performance.

Cheers,
Il 29/05/2015 15:20, Francois Gouget ha scritto:
> On Fri, 29 May 2015, Fabio Fantoni wrote:
> [...]
>> I'm trying the gst1 support but build fails with this error:
>> gstreamer_encoder.c: In function 'construct_pipeline':
>> gstreamer_encoder.c:219:41: error: 'GstEncoder' has no member named 'frc'
>>                       "deadline", encoder->frc.period * 1000 / 2,
>>
>> Why you tell that 1.0 is not functional? I suppose not only for the
>> build error I had.
> It's a merge bug. Sorry. I have pushed an update that fixes this so you 
> may want to rebase.
>
> The initial push had caps issues, mostly it was missing format=BGRx, 
> which caused GStreamer 1.0 not to work. Note that the failure mode 
> caused the video to be sent through the regular non-stream channel so it 
> looked like it worked.
>
> Now the GStreamer 1.0 situation is as follows:
>  * VP8 works.
>  * MJPEG fails because of a caps negotiation error despite the 
>    videoconvert element.
>  * H264 seems to work but the client fails to display it. So I'm not 
>    totally sure.
>
> Also I finally found the right parameters to improve the VP8 encoder 
> performance.
>
> Cheers,
>

Thanks, I tried vp8 after updating both server (with gst1) and client,
both software enc/dec on x86 image freeze don't happen but any video
(vlc, media player, flash ecc...) have critical artifact playing video
(difficult to explain).
I suppose that these warning I saw on client are related:
http://pastebin.com/GeTPyscG
> 0:00:00.150227452 27501      0x29f98f0 WARN            videodecoder
> gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec0> Received
> buffer without a new-segment. Assuming timestamps start from 0.
> 0:00:02.889876193 27501      0x29f9ed0 WARN            videodecoder
> gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec1> Received
> buffer without a new-segment. Assuming timestamps start from 0.


If you need more tests and/or informations tell me and I'll post them.

Thanks for any reply and sorry for my bad english.
On Mon, 1 Jun 2015, Fabio Fantoni wrote:
[...]
> I suppose that these warning I saw on client are related:
> http://pastebin.com/GeTPyscG
> > 0:00:00.150227452 27501      0x29f98f0 WARN            videodecoder
> > gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec0> Received
> > buffer without a new-segment. Assuming timestamps start from 0.
> > 0:00:02.889876193 27501      0x29f9ed0 WARN            videodecoder
> > gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec1> Received
> > buffer without a new-segment. Assuming timestamps start from 0.

It would be cleaner not to have these warnings (and I'll try to fix 
them), but I don't think they can cause display artifacts.

What kind of video application are you using for testing? Flash/YouTube? 
A local file with a media application like mplayer?
Il 02/06/2015 01:57, Francois Gouget ha scritto:
> On Mon, 1 Jun 2015, Fabio Fantoni wrote:
> [...]
>> I suppose that these warning I saw on client are related:
>> http://pastebin.com/GeTPyscG
>>> 0:00:00.150227452 27501      0x29f98f0 WARN            videodecoder
>>> gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec0> Received
>>> buffer without a new-segment. Assuming timestamps start from 0.
>>> 0:00:02.889876193 27501      0x29f9ed0 WARN            videodecoder
>>> gstvideodecoder.c:2026:gst_video_decoder_chain:<vp8dec1> Received
>>> buffer without a new-segment. Assuming timestamps start from 0.
> It would be cleaner not to have these warnings (and I'll try to fix 
> them), but I don't think they can cause display artifacts.
>
> What kind of video application are you using for testing? Flash/YouTube? 
> A local file with a media application like mplayer?

Video on youtube, hd trailer (file) with vlc (without vp8 works
correctly) and win media player.
I must post a small video of spice with artifacts and/or gst full debug
output?
On Tue, 2 Jun 2015, Fabio Fantoni wrote:
[...]
> Video on youtube,

Does it look like the video moves back and forth? Or maybe just the 
progress bar area, with some shearing too. That's a known issue caused 
by the progress bar popping up and down.


> hd trailer (file) with vlc (without vp8 works correctly) and win media 
> player.

I have not been able to really test with VLC here. I use Xspice which 
apparently does not support Xshm and that causes VLC to crash. And since 
I'm not testing with a VM windows media player is out.

Is bandwidth limited between the server and the client? With VP8, if an 
I frame gets dropped then the following P frames will be broken. I'm not 
sure how it handles that. Does adding error-resilient=1 or 
error-resilient=2 to vp8enc make a difference?
Il 02/06/2015 12:29, Francois Gouget ha scritto:
> On Tue, 2 Jun 2015, Fabio Fantoni wrote:
> [...]
>> Video on youtube,
> Does it look like the video moves back and forth? Or maybe just the 
> progress bar area, with some shearing too. That's a known issue caused 
> by the progress bar popping up and down.
>
>
>> hd trailer (file) with vlc (without vp8 works correctly) and win media 
>> player.
> I have not been able to really test with VLC here. I use Xspice which 
> apparently does not support Xshm and that causes VLC to crash. And since 
> I'm not testing with a VM windows media player is out.
>
> Is bandwidth limited between the server and the client? With VP8, if an 
> I frame gets dropped then the following P frames will be broken. I'm not 
> sure how it handles that. Does adding error-resilient=1 or 
> error-resilient=2 to vp8enc make a difference?
>

No bandwidth limit, on client I saw use between 1 and 3 mb, both server
and client have 1 gbps cabled network (lan).
Server used for testing is a dell PE T310 with cpu xeon X3450 is asus
K53S with core i5-2430M.
Server cpu usage of qemu process is over 300% on streming video.
spice streaming video setting is 'all' and codec 'gstreamer:vp8', no
error/warning in qemu logs.
I used github.com/fgouget/spice-gtk/commits/gst including "avoid
gstreamer deadlock" and github.com/fgouget/spice/commits/gst in latest test.
Client cpu and network usage seems low but streaming video is bad (with
vp8), here a small video example done with smartphone:
http://fantu.it/vari/spice-gtk-vp8.mp4

I'm going crazy, I'm not understand what can be the problem :(
Is more probable client or server side problem? What I must check better?

Thanks for any reply and sorry for my bad english.
On Thu, 4 Jun 2015, Fabio Fantoni wrote:
[...]
> No bandwidth limit, on client I saw use between 1 and 3 mb, both server
> and client have 1 gbps cabled network (lan).
> Server used for testing is a dell PE T310 with cpu xeon X3450 is asus
> K53S with core i5-2430M.
> Server cpu usage of qemu process is over 300% on streming video.
> spice streaming video setting is 'all' and codec 'gstreamer:vp8', no
> error/warning in qemu logs.
> I used github.com/fgouget/spice-gtk/commits/gst including "avoid
> gstreamer deadlock" and github.com/fgouget/spice/commits/gst in latest test.
> Client cpu and network usage seems low but streaming video is bad (with
> vp8), here a small video example done with smartphone:
> http://fantu.it/vari/spice-gtk-vp8.mp4

The video looks like there are two issues:

* The video corruption. It looks like an I frame was lost and the
  following P frames were thus decoded incorrectly. This is a bit 
  strange because in my experience VP8 throws the P frames in such a 
  case (so they are not displayed, unlike in h264's case).

* The short pause every few seconds. This may be related to the previous 
  issue: I frames are bigger than P frames. Maybe they take too long 
  coming across, thus forcing the client to pause waiting for them, and 
  maybe that's also why they sometimes get lost.

But given your network specifications that should not be the case. I 
have done tests with StreamingVideo=all here but it worked as well as 
when set to filter so that's probably not the source of the difference.

Overall I suspect more a problem on the server-side. Here's what you can 
do:

* First on the server check for messages that indicate some frames 
  were dropped like the ones below (SPICE_DEBUG_LEVEL=9):

  (../xf86-video-qxl/scripts/Xspice:31980): SpiceWorker-Debug **: 
    red_worker.c:3378:pre_stream_item_swap: ignoring drop, same pcg as 
    previous frame 296
  (../xf86-video-qxl/scripts/Xspice:31980): SpiceWorker-Debug **: 
    red_worker.c:10643:display_channel_release_item: not pushed (101)

  As far as I understand it's the second message that indicates a frame 
  has been dropped, but it systematically follows the first one so they 
  are related. These happen even when there's no bandwidth issue and 
  only happen in the GStreamer 1.0 case. And I have no idea what's 
  causing this so far :-(

* You can also look for 'server frame drop' in the server log. 
  Unlike the messages above, those would indicate a network problem.

* In gstreamer_encoder.c you may try adding a line like the following 
  near the start of adjust_bit_rate() to see if constraining the 
  bandwidth helps:
    encoder->bit_rate = 1024 * 1024 * 3;

  In the server log you can also look for 'base-bit-rate' and 'setting 
  the bit rate' to see what Spice and gstreamer_encoder think of the 
  bitrate respectively.

* Do you have the same issue if you use gstreamer:mjpeg on the 
  server? If gstreamer:mjpeg works fine then we'll know 
  it's not a general GStreamer issue but more a codec-specific one.

* The next thing to try would be gstreamer:h264. h264 is quite similar 
  to vp8 in caracteristics: it has I and P frames too and also uses 
  quite a bit of CPU, though it seems to handle threading a bit 
  better.

* Then if you can try running the server with GStreamer 0.10 this would 
  let us determine if it's a GStreamer version problem.

* The CPU usage on the server, 300%, does seem a bit high though for a 
  4C/8T CPU it should be manageable. Unfortunately gstreamer_encoder 
  already tunes vp8enc for speed. Maybe setting threads=4 or 5, or 
  cpu-used=16 can speed it up some more. For network usage maybe 
  max-intra-bitrate can help.