[Spice-devel,RFC,0.8,01/19] server/spice.h: semi-seamless migration interface, RHBZ #738266

Submitted by Yonit Halperin on Sept. 19, 2011, 4:46 p.m.

Details

Message ID 1316425632-3816-2-git-send-email-yhalperi@redhat.com
State New, archived
Headers show

Not browsing as part of any series.

Commit Message

Yonit Halperin Sept. 19, 2011, 4:46 p.m.
semi-seamless migration details:

migration source side
---------------------
(1) spice_server_migrate_info (*): tell client to link
    to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
    client_migrate_info cmd is asynchronous.
(2) Complete client_migrate_info only when the client has been connected
    to the target - wait for SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
(3) spice_server_migrate_end: tell client migration it can switch to the target - send
    SPICE_MSG_MAIN_MIGRATE_END.
(4) client cleans up all data related to the connection to the source and switches to the target.
    It sends SPICE_MSGC_MAIN_MIGRATE_END.

migration target side
---------------------
(1) the server identifies itself as a migraiton target since the client is linked with (connection_id != 0)
(2) server doesn't start the channels' logic (channel->link) till it receives SPICE_MSGC_MAIN_MIGRATE_END
    from the client.

*   After migration starts, the target qemu is blocked and cannot accept new spice client
    connections. Thus, we trigger the connection to the target upon client_migrate_info
    command.

Signed-off-by: Yonit Halperin <yhalperi@redhat.com>
---
 server/spice-experimental.h |    3 ---
 server/spice.h              |   21 ++++++++++++++++++++-
 2 files changed, 20 insertions(+), 4 deletions(-)

Patch hide | download patch | download mbox

diff --git a/server/spice-experimental.h b/server/spice-experimental.h
index 482ac44..6997aa0 100644
--- a/server/spice-experimental.h
+++ b/server/spice-experimental.h
@@ -29,16 +29,13 @@  void spice_server_net_wire_recv_packet(SpiceNetWireInstance *sin,
                                        const uint8_t *pkt, int len);
 
 /* spice seamless client migration (broken) */
-
 enum {
     SPICE_MIGRATE_CLIENT_NONE = 1,
     SPICE_MIGRATE_CLIENT_WAITING,
     SPICE_MIGRATE_CLIENT_READY,
 };
 
-int spice_server_migrate_start(SpiceServer *s);
 int spice_server_migrate_client_state(SpiceServer *s);
-int spice_server_migrate_end(SpiceServer *s, int completed);
 
 #endif // __SPICE_EXPERIMENTAL_H__
 
diff --git a/server/spice.h b/server/spice.h
index ac5a41e..8f7bcac 100644
--- a/server/spice.h
+++ b/server/spice.h
@@ -469,11 +469,30 @@  int spice_server_set_agent_copypaste(SpiceServer *s, int enable);
 int spice_server_get_sock_info(SpiceServer *s, struct sockaddr *sa, socklen_t *salen);
 int spice_server_get_peer_info(SpiceServer *s, struct sockaddr *sa, socklen_t *salen);
 
-/* spice switch-host client migration */
+/* migration interface */
+#define SPICE_INTERFACE_MIGRATION "migration"
+#define SPICE_INTERFACE_MIGRATION_MAJOR 1
+#define SPICE_INTERFACE_MIGRATION_MINOR 1
+typedef struct SpiceMigrateInterface SpiceMigrateInterface;
+typedef struct SpiceMigrateInstance SpiceMigrateInstance;
+typedef struct SpiceMigrateState SpiceMigrateState;
+
+struct SpiceMigrateInterface {
+    SpiceBaseInterface base;
+    void (*migrate_info_complete)(SpiceMigrateInstance *sin);
+};
 
+struct SpiceMigrateInstance {
+    SpiceBaseInstance base;
+    SpiceMigrateState *st;
+};
+
+/* spice switch-host client migration */
 int spice_server_migrate_info(SpiceServer *s, const char* dest,
                               int port, int secure_port,
                               const char* cert_subject);
 int spice_server_migrate_switch(SpiceServer *s);
 
+int spice_server_migrate_start(SpiceServer *s);
+int spice_server_migrate_end(SpiceServer *s, int completed);
 #endif

Comments

On Mon, Sep 19, 2011 at 12:46:54PM +0300, Yonit Halperin wrote:
> semi-seamless migration details:
> 
> migration source side
> ---------------------
> (1) spice_server_migrate_info (*): tell client to link
>     to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
>     client_migrate_info cmd is asynchronous.
> (2) Complete client_migrate_info only when the client has been connected
>     to the target - wait for SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
> (3) spice_server_migrate_end: tell client migration it can switch to the target - send
>     SPICE_MSG_MAIN_MIGRATE_END.
> (4) client cleans up all data related to the connection to the source and switches to the target.
>     It sends SPICE_MSGC_MAIN_MIGRATE_END.
> 
> migration target side
> ---------------------
> (1) the server identifies itself as a migraiton target since the client is linked with (connection_id != 0)
> (2) server doesn't start the channels' logic (channel->link) till it receives SPICE_MSGC_MAIN_MIGRATE_END
>     from the client.
> 
> *   After migration starts, the target qemu is blocked and cannot accept new spice client
>     connections. Thus, we trigger the connection to the target upon client_migrate_info
>     command.
> 
> Signed-off-by: Yonit Halperin <yhalperi@redhat.com>
> ---
>  server/spice-experimental.h |    3 ---
>  server/spice.h              |   21 ++++++++++++++++++++-
>  2 files changed, 20 insertions(+), 4 deletions(-)
> 
> diff --git a/server/spice-experimental.h b/server/spice-experimental.h
> index 482ac44..6997aa0 100644
> --- a/server/spice-experimental.h
> +++ b/server/spice-experimental.h
> @@ -29,16 +29,13 @@ void spice_server_net_wire_recv_packet(SpiceNetWireInstance *sin,
>                                         const uint8_t *pkt, int len);
>  
>  /* spice seamless client migration (broken) */
> -
>  enum {
>      SPICE_MIGRATE_CLIENT_NONE = 1,
>      SPICE_MIGRATE_CLIENT_WAITING,
>      SPICE_MIGRATE_CLIENT_READY,
>  };
>  
> -int spice_server_migrate_start(SpiceServer *s);
>  int spice_server_migrate_client_state(SpiceServer *s);
> -int spice_server_migrate_end(SpiceServer *s, int completed);
>  
>  #endif // __SPICE_EXPERIMENTAL_H__
>  
> diff --git a/server/spice.h b/server/spice.h
> index ac5a41e..8f7bcac 100644
> --- a/server/spice.h
> +++ b/server/spice.h
> @@ -469,11 +469,30 @@ int spice_server_set_agent_copypaste(SpiceServer *s, int enable);
>  int spice_server_get_sock_info(SpiceServer *s, struct sockaddr *sa, socklen_t *salen);
>  int spice_server_get_peer_info(SpiceServer *s, struct sockaddr *sa, socklen_t *salen);
>  
> -/* spice switch-host client migration */
> +/* migration interface */
> +#define SPICE_INTERFACE_MIGRATION "migration"
> +#define SPICE_INTERFACE_MIGRATION_MAJOR 1
> +#define SPICE_INTERFACE_MIGRATION_MINOR 1
> +typedef struct SpiceMigrateInterface SpiceMigrateInterface;
> +typedef struct SpiceMigrateInstance SpiceMigrateInstance;
> +typedef struct SpiceMigrateState SpiceMigrateState;
> +
> +struct SpiceMigrateInterface {
> +    SpiceBaseInterface base;
> +    void (*migrate_info_complete)(SpiceMigrateInstance *sin);

client_migrate_info_complete? like the monitor command it indicates the completion of.

> +};
>  
> +struct SpiceMigrateInstance {
> +    SpiceBaseInstance base;
> +    SpiceMigrateState *st;
> +};
> +
> +/* spice switch-host client migration */
>  int spice_server_migrate_info(SpiceServer *s, const char* dest,
>                                int port, int secure_port,
>                                const char* cert_subject);
>  int spice_server_migrate_switch(SpiceServer *s);
>  

+/* spice semi seamless client migration */ ?

> +int spice_server_migrate_start(SpiceServer *s);
> +int spice_server_migrate_end(SpiceServer *s, int completed);
>  #endif
> -- 
> 1.7.4.4
>
On 09/19/11 11:46, Yonit Halperin wrote:
> semi-seamless migration details:
>
> migration source side
> ---------------------
> (1) spice_server_migrate_info (*): tell client to link
>      to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
>      client_migrate_info cmd is asynchronous.
> (2) Complete client_migrate_info only when the client has been connected
>      to the target - wait for SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
> (3) spice_server_migrate_end: tell client migration it can switch to the target - send
>      SPICE_MSG_MAIN_MIGRATE_END.
> (4) client cleans up all data related to the connection to the source and switches to the target.
>      It sends SPICE_MSGC_MAIN_MIGRATE_END.

So the switch-host message will not be used any more?  How do you handle 
compatibility to spice servers which don't support this migration scheme?

This looks a bit like seamless migration half-done.  What is missing to 
full seamless migration support?  The spice channels need to transfer 
the state via client.  Also qemu on the source needs to wait for that to 
complete I guess?  Anything else?

>   enum {
>       SPICE_MIGRATE_CLIENT_NONE = 1,
>       SPICE_MIGRATE_CLIENT_WAITING,
>       SPICE_MIGRATE_CLIENT_READY,
>   };

Is that still needed?

> +/* migration interface */
> +#define SPICE_INTERFACE_MIGRATION "migration"
> +#define SPICE_INTERFACE_MIGRATION_MAJOR 1
> +#define SPICE_INTERFACE_MIGRATION_MINOR 1
> +typedef struct SpiceMigrateInterface SpiceMigrateInterface;
> +typedef struct SpiceMigrateInstance SpiceMigrateInstance;
> +typedef struct SpiceMigrateState SpiceMigrateState;
> +
> +struct SpiceMigrateInterface {
> +    SpiceBaseInterface base;
> +    void (*migrate_info_complete)(SpiceMigrateInstance *sin);
> +};
>
> +struct SpiceMigrateInstance {
> +    SpiceBaseInstance base;
> +    SpiceMigrateState *st;
> +};

Why a new interface?  The only purpose it seems to serve is adding a 
single callback.  It is registered unconditionally by the spice core 
code in qemu.  I fail to see why we can't just add the callback to the 
core interface.  That would save a bunch of code to handle the new 
interface.

I think we should try to make sure the qemu <-> spice-server migration 
API can handle both semi-seamless and real seamless client migration. 
Then the two spice servers can negotiate themself which method they 
pick, without qemu being involved here.  I think the minimum we'll need 
here is a second notification (when spice channel state migration via 
slient is finished), either using a second callback or by adding a 
reason argument to the one we have.  But maybe just blocking in 
spice_server_migrate_end() works too (the guest is stopped at that point 
anyway, so going async doesn't buy us that much here ...).

What is the plan for the master branch?  How will this work with 
multiclient enabled?

cheers,
   Gerd
On 09/19/2011 03:15 PM, Gerd Hoffmann wrote:
> On 09/19/11 11:46, Yonit Halperin wrote:
>> semi-seamless migration details:
>>
>> migration source side
>> ---------------------
>> (1) spice_server_migrate_info (*): tell client to link
>> to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
>> client_migrate_info cmd is asynchronous.
>> (2) Complete client_migrate_info only when the client has been connected
>> to the target - wait for
>> SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
>> (3) spice_server_migrate_end: tell client migration it can switch to
>> the target - send
>> SPICE_MSG_MAIN_MIGRATE_END.
>> (4) client cleans up all data related to the connection to the source
>> and switches to the target.
>> It sends SPICE_MSGC_MAIN_MIGRATE_END.
>
> So the switch-host message will not be used any more? How do you handle
> compatibility to spice servers which don't support this migration scheme?
>
Older qemu/spice-server will still use the switch host message and the 
spice client still support it.
With respect to spice-severs that doesn't support this scheme:
qemu checks #ifdef SPICE_INTERFACE_MIGRATION, and acts accordingly 
(calls switch-host or migrate_end).
The spice client responds according to the server:
if <=Rhel6.0 : seamless migration
if Rhel6.1: Switch host
if Rhel6.2: semi-seamless

Each scheme has its own code pathway in the client.

> This looks a bit like seamless migration half-done. What is missing to
> full seamless migration support? The spice channels need to transfer the
> state via client. Also qemu on the source needs to wait for that to
> complete I guess? Anything else?
We are still missing keeping the target stopped till the target spice 
server is restored from the data passed through the client.
Thanks to the fix Alon sent for RHBZ #729621, we could block the target 
from starting using a vm state change handler, but it is a workaround...
IMHO, if we could have async notifiers for migration before it starts 
and just before it completes (as we had in Rhel5), that would have been 
the best solution (it will also allow us not to use client_migrate_info 
for the pre-connection to the target). By async notifiers I mean
(1) notifiers that hold the migration process till they complete.
(2) to add notifications (a) before migration starts (today we receive 
one only after it starts), (b) before migration completes - when the src 
and target are both stopped (today we don't have control over the target 
state).

>
>> enum {
>> SPICE_MIGRATE_CLIENT_NONE = 1,
>> SPICE_MIGRATE_CLIENT_WAITING,
>> SPICE_MIGRATE_CLIENT_READY,
>> };
>
> Is that still needed?
>
Don't think so.

>> +/* migration interface */
>> +#define SPICE_INTERFACE_MIGRATION "migration"
>> +#define SPICE_INTERFACE_MIGRATION_MAJOR 1
>> +#define SPICE_INTERFACE_MIGRATION_MINOR 1
>> +typedef struct SpiceMigrateInterface SpiceMigrateInterface;
>> +typedef struct SpiceMigrateInstance SpiceMigrateInstance;
>> +typedef struct SpiceMigrateState SpiceMigrateState;
>> +
>> +struct SpiceMigrateInterface {
>> + SpiceBaseInterface base;
>> + void (*migrate_info_complete)(SpiceMigrateInstance *sin);
>> +};
>>
>> +struct SpiceMigrateInstance {
>> + SpiceBaseInstance base;
>> + SpiceMigrateState *st;
>> +};
>
> Why a new interface? The only purpose it seems to serve is adding a
> single callback. It is registered unconditionally by the spice core code
> in qemu. I fail to see why we can't just add the callback to the core
> interface. That would save a bunch of code to handle the new interface.
>
> I think we should try to make sure the qemu <-> spice-server migration
> API can handle both semi-seamless and real seamless client migration.
I added a new interface since I think that for supporting seamless 
migration we may need more callbacks (e.g., if the above async notifiers 
will be implemented). IMHO, it is more readable then adding it to the 
core interface.
> Then the two spice servers can negotiate themself which method they
> pick, without qemu being involved here. I think the minimum we'll need
> here is a second notification (when spice channel state migration via
> slient is finished), either using a second callback or by adding a
> reason argument to the one we have. But maybe just blocking in
> spice_server_migrate_end() works too (the guest is stopped at that point
> anyway, so going async doesn't buy us that much here ...).
>
> What is the plan for the master branch? How will this work with
> multiclient enabled?
>
For one client - the same.
For multiclient - not sure yet, maybe for small number of clients do the 
same, and for the rest fall back to switch_host? (with the risk of 
losing the ticket).
> cheers,
> Gerd
>
On Mon, Sep 19, 2011 at 04:06:25PM +0300, Yonit Halperin wrote:
> On 09/19/2011 03:15 PM, Gerd Hoffmann wrote:
> >On 09/19/11 11:46, Yonit Halperin wrote:
> >>semi-seamless migration details:
> >>
> >>migration source side
> >>---------------------
> >>(1) spice_server_migrate_info (*): tell client to link
> >>to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
> >>client_migrate_info cmd is asynchronous.
> >>(2) Complete client_migrate_info only when the client has been connected
> >>to the target - wait for
> >>SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
> >>(3) spice_server_migrate_end: tell client migration it can switch to
> >>the target - send
> >>SPICE_MSG_MAIN_MIGRATE_END.
> >>(4) client cleans up all data related to the connection to the source
> >>and switches to the target.
> >>It sends SPICE_MSGC_MAIN_MIGRATE_END.
> >
> >So the switch-host message will not be used any more? How do you handle
> >compatibility to spice servers which don't support this migration scheme?
> >
> Older qemu/spice-server will still use the switch host message and
> the spice client still support it.
> With respect to spice-severs that doesn't support this scheme:
> qemu checks #ifdef SPICE_INTERFACE_MIGRATION, and acts accordingly
> (calls switch-host or migrate_end).
> The spice client responds according to the server:
> if <=Rhel6.0 : seamless migration
> if Rhel6.1: Switch host
> if Rhel6.2: semi-seamless

We need to handle another case:
 New client.
 New source server.
 Old target server.

Server and client both have the SEMI_SEAMLESS cap, so server uses semi-seamless,
client connects to target, but target doesn't support semi seamless and proceeds
to send updates as usual without waiting for a SPICEC_MSG_MIGRATE_END.

Two possible solutions:
 1. client_migrate_info - add an extra parameter for "target version". Spice server will
 see the target is too old, and use switch host. Requires change to libvirt, so
 ignored for now.
 2. When client connects to the server it sees that the server doesn't have the SEMI_SEAMLESS_CAP
 present. So it disconnects, and sends to the source a SPICEC_MSG_MIGRATE_TARGET_CONNECTION_FAILED_TARGET_TOO_OLD
, or maybe just a generic error message (if we have something like that) and use that as the reason.
 The server sees this, and proceeds to do the old switch host behavior. This doesn're require libvirt changes,
 it does create two connections to the target, but at least the vm is never stopped during that time.

> 
> Each scheme has its own code pathway in the client.
> 
> >This looks a bit like seamless migration half-done. What is missing to
> >full seamless migration support? The spice channels need to transfer the
> >state via client. Also qemu on the source needs to wait for that to
> >complete I guess? Anything else?
> We are still missing keeping the target stopped till the target
> spice server is restored from the data passed through the client.
> Thanks to the fix Alon sent for RHBZ #729621, we could block the
> target from starting using a vm state change handler, but it is a
> workaround...
> IMHO, if we could have async notifiers for migration before it
> starts and just before it completes (as we had in Rhel5), that would
> have been the best solution (it will also allow us not to use
> client_migrate_info for the pre-connection to the target). By async
> notifiers I mean
> (1) notifiers that hold the migration process till they complete.
> (2) to add notifications (a) before migration starts (today we
> receive one only after it starts), (b) before migration completes -
> when the src and target are both stopped (today we don't have
> control over the target state).
> 
> >
> >>enum {
> >>SPICE_MIGRATE_CLIENT_NONE = 1,
> >>SPICE_MIGRATE_CLIENT_WAITING,
> >>SPICE_MIGRATE_CLIENT_READY,
> >>};
> >
> >Is that still needed?
> >
> Don't think so.
> 
> >>+/* migration interface */
> >>+#define SPICE_INTERFACE_MIGRATION "migration"
> >>+#define SPICE_INTERFACE_MIGRATION_MAJOR 1
> >>+#define SPICE_INTERFACE_MIGRATION_MINOR 1
> >>+typedef struct SpiceMigrateInterface SpiceMigrateInterface;
> >>+typedef struct SpiceMigrateInstance SpiceMigrateInstance;
> >>+typedef struct SpiceMigrateState SpiceMigrateState;
> >>+
> >>+struct SpiceMigrateInterface {
> >>+ SpiceBaseInterface base;
> >>+ void (*migrate_info_complete)(SpiceMigrateInstance *sin);
> >>+};
> >>
> >>+struct SpiceMigrateInstance {
> >>+ SpiceBaseInstance base;
> >>+ SpiceMigrateState *st;
> >>+};
> >
> >Why a new interface? The only purpose it seems to serve is adding a
> >single callback. It is registered unconditionally by the spice core code
> >in qemu. I fail to see why we can't just add the callback to the core
> >interface. That would save a bunch of code to handle the new interface.
> >
> >I think we should try to make sure the qemu <-> spice-server migration
> >API can handle both semi-seamless and real seamless client migration.
> I added a new interface since I think that for supporting seamless
> migration we may need more callbacks (e.g., if the above async
> notifiers will be implemented). IMHO, it is more readable then
> adding it to the core interface.
> >Then the two spice servers can negotiate themself which method they
> >pick, without qemu being involved here. I think the minimum we'll need
> >here is a second notification (when spice channel state migration via
> >slient is finished), either using a second callback or by adding a
> >reason argument to the one we have. But maybe just blocking in
> >spice_server_migrate_end() works too (the guest is stopped at that point
> >anyway, so going async doesn't buy us that much here ...).
> >
> >What is the plan for the master branch? How will this work with
> >multiclient enabled?
> >
> For one client - the same.
> For multiclient - not sure yet, maybe for small number of clients do
> the same, and for the rest fall back to switch_host? (with the risk
> of losing the ticket).
> >cheers,
> >Gerd
> >
>
Hi,

> We need to handle another case: New client. New source server. Old
> target server.

Indeed.

> 2. When client connects to the server it sees that the server doesn't
> have the SEMI_SEAMLESS_CAP present. So it disconnects, and sends to
> the source a
> SPICEC_MSG_MIGRATE_TARGET_CONNECTION_FAILED_TARGET_TOO_OLD , or maybe
> just a generic error message (if we have something like that) and use
> that as the reason. The server sees this, and proceeds to do the old
> switch host behavior.

Sounds good to me.

>>> This looks a bit like seamless migration half-done. What is
>>> missing to full seamless migration support? The spice channels
>>> need to transfer the state via client. Also qemu on the source
>>> needs to wait for that to complete I guess? Anything else?

>> We are still missing keeping the target stopped till the target
>> spice server is restored from the data passed through the client.
>> Thanks to the fix Alon sent for RHBZ #729621, we could block the
>> target from starting using a vm state change handler, but it is a
>> workaround... IMHO, if we could have async notifiers for migration
>> before it starts and just before it completes (as we had in Rhel5),
>> that would have been the best solution (it will also allow us not
>> to use client_migrate_info for the pre-connection to the target).

For now I'll try to see the big picture (aka API needed), not the little 
implementation details.  Whenever we kick off the migration from the 
client_info or the migration notifier doesn't matter for the 
libspice-server API, that is an detail which can easily be changed once 
we have the notifiers in place.

The steps needed for seamless migration, qemu on the source:

   (1) spice_server_migrate_info()

   (2) spice_server_migrate_start()

   (3) wait for the callback, async needed as the vm is running.

   (4) do migration, live stage.

   (5) stop virtual machine (and qxl worker).

   (6) Guess we need to notify spice-server here?
       Maybe call spice_server_migrate_start() here and rename the
       call in (2) to spice_server_migrate_connect() ?

   (7) do migration, non-live stage.

   (8) spice_server_migrate_end().

Qemu on the target:

   (1) receive migration

   (2) wait for spice-server to complete client migration
       [ need api for that ]

   (3) start virtual machine (and qxl worker).

Correct?

semi-seamless should work with that workflow too, even though 
spice-server probably hasn't anything to do at source step (6) and 
target step (2), correct?

cheers,
   Gerd
On 09/19/2011 04:24 PM, Alon Levy wrote:
> On Mon, Sep 19, 2011 at 04:06:25PM +0300, Yonit Halperin wrote:
>> On 09/19/2011 03:15 PM, Gerd Hoffmann wrote:
>>> On 09/19/11 11:46, Yonit Halperin wrote:
>>>> semi-seamless migration details:
>>>>
>>>> migration source side
>>>> ---------------------
>>>> (1) spice_server_migrate_info (*): tell client to link
>>>> to the target side - send SPICE_MSG_MAIN_MIGRATE_BEGIN.
>>>> client_migrate_info cmd is asynchronous.
>>>> (2) Complete client_migrate_info only when the client has been connected
>>>> to the target - wait for
>>>> SPICE_MSGC_MAIN_MIGRATE_(CONNECTED|CONNECT_ERROR) or a timeout.
>>>> (3) spice_server_migrate_end: tell client migration it can switch to
>>>> the target - send
>>>> SPICE_MSG_MAIN_MIGRATE_END.
>>>> (4) client cleans up all data related to the connection to the source
>>>> and switches to the target.
>>>> It sends SPICE_MSGC_MAIN_MIGRATE_END.
>>>
>>> So the switch-host message will not be used any more? How do you handle
>>> compatibility to spice servers which don't support this migration scheme?
>>>
>> Older qemu/spice-server will still use the switch host message and
>> the spice client still support it.
>> With respect to spice-severs that doesn't support this scheme:
>> qemu checks #ifdef SPICE_INTERFACE_MIGRATION, and acts accordingly
>> (calls switch-host or migrate_end).
>> The spice client responds according to the server:
>> if<=Rhel6.0 : seamless migration
>> if Rhel6.1: Switch host
>> if Rhel6.2: semi-seamless
>
> We need to handle another case:
>   New client.
>   New source server.
>   Old target server.
>
> Server and client both have the SEMI_SEAMLESS cap, so server uses semi-seamless,
> client connects to target, but target doesn't support semi seamless and proceeds
> to send updates as usual without waiting for a SPICEC_MSG_MIGRATE_END.
>
> Two possible solutions:
>   1. client_migrate_info - add an extra parameter for "target version". Spice server will
>   see the target is too old, and use switch host. Requires change to libvirt, so
>   ignored for now.
>   2. When client connects to the server it sees that the server doesn't have the SEMI_SEAMLESS_CAP
>   present. So it disconnects, and sends to the source a SPICEC_MSG_MIGRATE_TARGET_CONNECTION_FAILED_TARGET_TOO_OLD
> , or maybe just a generic error message (if we have something like that) and use that as the reason.
>   The server sees this, and proceeds to do the old switch host behavior. This doesn're require libvirt changes,
>   it does create two connections to the target, but at least the vm is never stopped during that time.
>
Actually the connection to the old target server will fail anyway with 
SPICE_LINK_ERR_BAD_CONNECTION_ID (*) and the client will send 
SPICE_MSGC_MAIN_MIGRATE_CONNECT_ERROR. So I guess the only change needed 
is to fall back to switch host when 
SPICE_MSGC_MAIN_MIGRATE_CONNECT_ERROR happens (currently the client is 
disconnected).

(*) the client sends a connection_id != 0 to the target server. The 
target server compares it to reds->link_id, which is 0 (in RHEL5 we used 
to send the connection id from the src to the target), and rejects the 
connection.


>>
>> Each scheme has its own code pathway in the client.
>>
>>> This looks a bit like seamless migration half-done. What is missing to
>>> full seamless migration support? The spice channels need to transfer the
>>> state via client. Also qemu on the source needs to wait for that to
>>> complete I guess? Anything else?
>> We are still missing keeping the target stopped till the target
>> spice server is restored from the data passed through the client.
>> Thanks to the fix Alon sent for RHBZ #729621, we could block the
>> target from starting using a vm state change handler, but it is a
>> workaround...
>> IMHO, if we could have async notifiers for migration before it
>> starts and just before it completes (as we had in Rhel5), that would
>> have been the best solution (it will also allow us not to use
>> client_migrate_info for the pre-connection to the target). By async
>> notifiers I mean
>> (1) notifiers that hold the migration process till they complete.
>> (2) to add notifications (a) before migration starts (today we
>> receive one only after it starts), (b) before migration completes -
>> when the src and target are both stopped (today we don't have
>> control over the target state).
>>
>>>
>>>> enum {
>>>> SPICE_MIGRATE_CLIENT_NONE = 1,
>>>> SPICE_MIGRATE_CLIENT_WAITING,
>>>> SPICE_MIGRATE_CLIENT_READY,
>>>> };
>>>
>>> Is that still needed?
>>>
>> Don't think so.
>>
>>>> +/* migration interface */
>>>> +#define SPICE_INTERFACE_MIGRATION "migration"
>>>> +#define SPICE_INTERFACE_MIGRATION_MAJOR 1
>>>> +#define SPICE_INTERFACE_MIGRATION_MINOR 1
>>>> +typedef struct SpiceMigrateInterface SpiceMigrateInterface;
>>>> +typedef struct SpiceMigrateInstance SpiceMigrateInstance;
>>>> +typedef struct SpiceMigrateState SpiceMigrateState;
>>>> +
>>>> +struct SpiceMigrateInterface {
>>>> + SpiceBaseInterface base;
>>>> + void (*migrate_info_complete)(SpiceMigrateInstance *sin);
>>>> +};
>>>>
>>>> +struct SpiceMigrateInstance {
>>>> + SpiceBaseInstance base;
>>>> + SpiceMigrateState *st;
>>>> +};
>>>
>>> Why a new interface? The only purpose it seems to serve is adding a
>>> single callback. It is registered unconditionally by the spice core code
>>> in qemu. I fail to see why we can't just add the callback to the core
>>> interface. That would save a bunch of code to handle the new interface.
>>>
>>> I think we should try to make sure the qemu<->  spice-server migration
>>> API can handle both semi-seamless and real seamless client migration.
>> I added a new interface since I think that for supporting seamless
>> migration we may need more callbacks (e.g., if the above async
>> notifiers will be implemented). IMHO, it is more readable then
>> adding it to the core interface.
>>> Then the two spice servers can negotiate themself which method they
>>> pick, without qemu being involved here. I think the minimum we'll need
>>> here is a second notification (when spice channel state migration via
>>> slient is finished), either using a second callback or by adding a
>>> reason argument to the one we have. But maybe just blocking in
>>> spice_server_migrate_end() works too (the guest is stopped at that point
>>> anyway, so going async doesn't buy us that much here ...).
>>>
>>> What is the plan for the master branch? How will this work with
>>> multiclient enabled?
>>>
>> For one client - the same.
>> For multiclient - not sure yet, maybe for small number of clients do
>> the same, and for the rest fall back to switch_host? (with the risk
>> of losing the ticket).
>>> cheers,
>>> Gerd
>>>
>>
On 09/19/2011 07:13 PM, Gerd Hoffmann wrote:
>   Hi,
>
>> We need to handle another case: New client. New source server. Old
>> target server.
>
> Indeed.
>
>> 2. When client connects to the server it sees that the server doesn't
>> have the SEMI_SEAMLESS_CAP present. So it disconnects, and sends to
>> the source a
>> SPICEC_MSG_MIGRATE_TARGET_CONNECTION_FAILED_TARGET_TOO_OLD , or maybe
>> just a generic error message (if we have something like that) and use
>> that as the reason. The server sees this, and proceeds to do the old
>> switch host behavior.
>
> Sounds good to me.
>
>>>> This looks a bit like seamless migration half-done. What is
>>>> missing to full seamless migration support? The spice channels
>>>> need to transfer the state via client. Also qemu on the source
>>>> needs to wait for that to complete I guess? Anything else?
>
>>> We are still missing keeping the target stopped till the target
>>> spice server is restored from the data passed through the client.
>>> Thanks to the fix Alon sent for RHBZ #729621, we could block the
>>> target from starting using a vm state change handler, but it is a
>>> workaround... IMHO, if we could have async notifiers for migration
>>> before it starts and just before it completes (as we had in Rhel5),
>>> that would have been the best solution (it will also allow us not
>>> to use client_migrate_info for the pre-connection to the target).
>
> For now I'll try to see the big picture (aka API needed), not the little
> implementation details. Whenever we kick off the migration from the
> client_info or the migration notifier doesn't matter for the
> libspice-server API, that is an detail which can easily be changed once
> we have the notifiers in place.
>
> The steps needed for seamless migration, qemu on the source:
>
> (1) spice_server_migrate_info()
>
> (2) spice_server_migrate_start()
>
> (3) wait for the callback, async needed as the vm is running.
>
> (4) do migration, live stage.
>
> (5) stop virtual machine (and qxl worker).
>
> (6) Guess we need to notify spice-server here?
> Maybe call spice_server_migrate_start() here and rename the
> call in (2) to spice_server_migrate_connect() ?
>
> (7) do migration, non-live stage.
>
> (8) spice_server_migrate_end().
>

(A) Is it possible to hold the target paused (but not blocked) after the 
non-live stage completes? If it is possible (6) is not necessary as long 
as spice_server_migrate_end will be called before the target vm is started.


> Qemu on the target:
>
> (1) receive migration
>
> (2) wait for spice-server to complete client migration
> [ need api for that ]
>
(B) If the answer for (A) is yes, maybe this should be done through the 
client and the src server:
(1) target spice-server completes client migration, acks the client,
(2) the client disconnects from the src or sends an appropriate message 
to the src server
(3) src server completes spice_server_migrate_end (async?), and make the 
target vm start.

If the answer for (A) is no, then spice will need qemu to have a 
migration completion async notifier in the target side as well.

> (3) start virtual machine (and qxl worker).
>
> Correct?
>
> semi-seamless should work with that workflow too, even though
> spice-server probably hasn't anything to do at source step (6) and
> target step (2), correct?
Yes
>
> cheers,
> Gerd
>
>> The steps needed for seamless migration, qemu on the source:
>>
>> (1) spice_server_migrate_info()
>>
>> (2) spice_server_migrate_start()
>>
>> (3) wait for the callback, async needed as the vm is running.
>>
>> (4) do migration, live stage.
>>
>> (5) stop virtual machine (and qxl worker).
>>
>> (6) Guess we need to notify spice-server here?
>> Maybe call spice_server_migrate_start() here and rename the
>> call in (2) to spice_server_migrate_connect() ?
>>
>> (7) do migration, non-live stage.
>>
>> (8) spice_server_migrate_end().
>>
>
> (A) Is it possible to hold the target paused (but not blocked) after the
> non-live stage completes?

That is on the list below for the target qemu ;)

We will need that no matter what, we might have to create some 
infrastructure for that though.  I think this pretty much implies that 
the target qemu needs to tell the spice server somehow whenever it can 
handle this (and thus support seamless migration), so spice-server can 
set the capabilities on the main channel accordingly.

> If it is possible (6) is not necessary as long
> as spice_server_migrate_end will be called before the target vm is started.

But I think it will help keeping the vm downtime low because the final 
stage of vm migration and spice seamless migration can run in parallel. 
  At least the display channel which runs in its own thread should 
benefit from that.  For the other channels it probably isn't a big 
difference today due to qemu not listening on the sockets.  That might 
change in the future though.

>> Qemu on the target:
>>
>> (1) receive migration
>>
>> (2) wait for spice-server to complete client migration
>> [ need api for that ]
>>
> (B) If the answer for (A) is yes, maybe this should be done through the
> client and the src server:
> (1) target spice-server completes client migration, acks the client,
> (2) the client disconnects from the src or sends an appropriate message
> to the src server
> (3) src server completes spice_server_migrate_end (async?), and make the
> target vm start.
>
> If the answer for (A) is no, then spice will need qemu to have a
> migration completion async notifier in the target side as well.

Oh, I think I misunderstood your quesion above.  I think we should 
handle this completely on the target side, i.e. have qemu wait for both 
vm migration and spice client migration finish before re-starting the 
guest (leaving aside for now how we are going to actually implement 
that). That should be more robust than doing some kind of syncronization 
using the client.  There is no need to handle icky corner cases like the 
client just exiting the moment you want use it to send a notification to 
the other qemu.

cheers,
   Gerd