player: add display sync mode

wm4 · wm4 · commit 031555fbe6cf · 2015-08-10T18:48:45.000+02:00
If this mode is enabled, the player tries to strictly synchronize video
to display refresh. It will adjust playback speed to match the display,
so if you play 23.976 fps video on a 24 Hz screen, playback speed is
increased by approximately 1/1000. Audio wll be resampled to keep up
with playback.

This is different from the default sync mode, which will sync video to
audio, with the consequence that video might skip or repeat a frame once
in a while to make video keep up with audio.

This is still unpolished. There are some major problems as well; in
particular, mkv VFR files won't work well. The reason is that Matroska
is terrible and rounds timestamps to milliseconds. This makes it rather
hard to guess the framerate of a section of video that is playing. We
could probably fix this by just accepting jittery timestamps (instead
of explicitly disabling the sync code in this case), but I'm not ready
to accept such a solution yet.

Another issue is that we are extremely reliant on OS video and audio
APIs working in an expected manner, which of course is not too often
the case. Consequently, the new sync mode is a bit fragile.
diff --git a/DOCS/interface-changes.rst b/DOCS/interface-changes.rst
@@ -20,7 +20,10 @@ Interface changes
 ::
 
  --- mpv 0.10.0 will be released ---
-    - add "audio-speed-correction" and "video-speed-correction" properties
+    - add --video-sync* options
+      "display-sync-active" property
+      "vo-missed-frame-count" property
+      "audio-speed-correction" and "video-speed-correction" properties
     - remove --demuxer-readahead-packets and --demuxer-readahead-bytes
       add --demuxer-max-packets and --demuxer-max-bytes
       (the new options are not replacement and have very different semantics)
diff --git a/DOCS/man/input.rst b/DOCS/man/input.rst
@@ -830,6 +830,12 @@ Property list
     Factor multiplied with ``speed`` at which the player attempts to play the
     file. Usually it's exactly 1. (Display sync mode will make this useful.)
 
+    OSD formatting will display it in the form of ``+1.23456%``, with the number
+    being ``(raw - 1) * 100`` for the given raw property value.
+
+``display-sync-active``
+    Return whether ``--video-sync=display`` is actually active.
+
 ``filename``
     Currently played file, with path stripped. If this is an URL, try to undo
     percent encoding as well. (The result is not necessarily correct, but
diff --git a/DOCS/man/mpv.rst b/DOCS/man/mpv.rst
@@ -526,6 +526,13 @@ listed.
   if there is audio "missing", or not enough frames can be dropped. Usually
   this will indicate a problem. (``total-avsync-change`` property.)
 - Encoding state in ``{...}``, only shown in encoding mode.
+- Display sync state. If display sync is active (``display-sync-active``
+  property), this shows ``DS: 1.002``, where the number is the speed change
+  factor applied to audio to achieve sync to display (``audio-speed-correction``
+  property). In sync modes which don't resample, this will always be ``1.000``.
+- Missed frames, e.g. ``Missed: 4``. (``vo-missed-frame-count`` property.) Shows
+  up in display sync mode only. This is incremented each time a frame took
+  longer to display than intended.
 - Dropped frames, e.g. ``Dropped: 4``. Shows up only if the count is not 0. Can
   grow if the video framerate is higher than that of the display, or if video
   rendering is too slow. Also can be incremented on "hiccups" and when the video
diff --git a/DOCS/man/options.rst b/DOCS/man/options.rst
@@ -3363,6 +3363,72 @@ Miscellaneous
     out. This delay in reaction time to sudden A/V offsets should be the only
     side-effect of turning this option on, for all sound drivers.
 
+``--video-sync=<audio|...>``
+    How the player synchronizes audio and video.
+
+    The modes starting with ``display-`` try to output video frames completely
+    synchronously to the display, using the detected display vertical refresh
+    rate as a hint how fast frames will be displayed on average. These modes
+    change video speed slightly to match the display. See ``--video-sync-...``
+    options for fine tuning. The robustness of this mode is further reduced by
+    making a some idealized assumptions, which may not always apply in reality.
+    Behavior can depend on the VO and the system's video and audio drivers.
+    Media files must use constant framerate. Section-wise VFR might work as well
+    with some container formats (but not e.g. mkv). If the sync code detects
+    severe A/V desync, or the framerate cannot be detected, the player
+    automatically reverts to ``audio`` mode for some time or permanently.
+
+    The modes with ``desync`` in their names do not attempt to keep audio/video
+    in sync. They will slowly (or quickly) desync, until e.g. the next seek
+    happens. These modes are meant for testing, not serious use.
+
+    :audio:             Time video frames to audio. This is the most robust
+                        mode, because the player doesn't have to assume anything
+                        about how the display behaves. The disadvantage is that
+                        it can lead to occasional frame drops or repeats. If
+                        audio is disabled, this uses the system clock. This is
+                        the default mode.
+    :display-resample:  Resample audio to match the video. This mode will also
+                        try to adjust audio speed to compensate for other drift.
+                        (This means it will play the audio at a different speed
+                        every once in a while to reduce the A/V difference.)
+    :display-resample-vdrop:  Resample audio to match the video. Drop video
+                        frames to compensate for drift.
+    :display-resample-desync: Like the previous mode, but no A/V compensation.
+    :display-vdrop:     Drop or repeat video frames to compensate desyncing
+                        video. (Although it should have the same effects as
+                        ``audio``, the implementation is very different.)
+    :display-desync:    Sync video to display, and let audio play on its own.
+    :desync:            Sync video according to system clock, and let audio play
+                        on its own.
+
+``--video-sync-max-video-change=<value>``
+    Maximum speed difference in percent that is applied to video with
+    ``--video-sync=display-...`` (default: 1). Display sync mode will be
+    disabled if the monitor and video refresh way do not match within the
+    given range. It tries multiples as well: playing 30 fps video on a 60 Hz
+    screen will duplicate every second frame. Playing 24 fps video on a 60 Hz
+    screen will play video in a 2-3-2-3-... pattern.
+
+    The default settings are not loose enough to speed up 23.976 fps video to
+    25 fps. We consider the pitch change too extreme to allow this behavior
+    by default. Set this option to a value of ``5`` to enable it.
+
+    Note that in the ``--video-sync=display-resample`` mode, audio speed will
+    additionally be changed by a small amount if necessary for A/V sync. See
+    ``--video-sync-max-audio-change``.
+
+``--video-sync-max-video-change=<value>``
+    Maximum *additional* speed difference in percent that is applied to audio
+    with ``--video-sync=display-...`` (default: 0.125). Normally, the player
+    play the audio at the speed of the video. But if the difference between
+    audio and video position is too high, e.g. due to drift or other timing
+    errors, it will attempt to speed up or slow down audio by this additional
+    factor. Too low values could lead to video frame dropping or repeating if
+    the A/V desync cannot be compensated, too high values could lead to chaotic
+    frame dropping due to the audio "overshooting" and skipping multiple video
+    frames before the sync logic can react.
+
 ``--mf-fps=<value>``
     Framerate used when decoding from multiple PNG or JPEG files with ``mf://``
     (default: 1).
diff --git a/options/options.c b/options/options.c
@@ -518,6 +518,18 @@ const m_option_t mp_opts[] = {
     OPT_CHOICE("pts-association-mode", user_pts_assoc_mode, 0,
                ({"auto", 0}, {"decoder", 1}, {"sort", 2})),
     OPT_FLAG("initial-audio-sync", initial_audio_sync, 0),
+    OPT_CHOICE("video-sync", video_sync, 0,
+               ({"audio", VS_DEFAULT},
+                {"display-resample", VS_DISP_RESAMPLE},
+                {"display-resample-vdrop", VS_DISP_RESAMPLE_VDROP},
+                {"display-resample-desync", VS_DISP_RESAMPLE_NONE},
+                {"display-vdrop", VS_DISP_VDROP},
+                {"display-desync", VS_DISP_NONE},
+                {"desync", VS_NONE})),
+    OPT_DOUBLE("video-sync-max-video-change", sync_max_video_change,
+               M_OPT_MIN, .min = 0),
+    OPT_DOUBLE("video-sync-max-audio-change", sync_max_audio_change,
+               M_OPT_MIN | M_OPT_MAX, .min = 0, .max = 1),
     OPT_CHOICE("hr-seek", hr_seek, 0,
                ({"no", -1}, {"absolute", 0}, {"yes", 1}, {"always", 1})),
     OPT_FLOAT("hr-seek-demuxer-offset", hr_seek_demuxer_offset, 0),
@@ -710,6 +722,8 @@ const struct MPOpts mp_default_opts = {
     .chapter_merge_threshold = 100,
     .chapter_seek_threshold = 5.0,
     .hr_seek_framedrop = 1,
+    .sync_max_video_change = 1,
+    .sync_max_audio_change = 0.125,
     .load_config = 1,
     .position_resume = 1,
     .stream_cache = {
diff --git a/options/options.h b/options/options.h
@@ -144,6 +144,9 @@ typedef struct MPOpts {
     int correct_pts;
     int user_pts_assoc_mode;
     int initial_audio_sync;
+    int video_sync;
+    double sync_max_video_change;
+    double sync_max_audio_change;
     int hr_seek;
     float hr_seek_demuxer_offset;
     int hr_seek_framedrop;
diff --git a/player/command.c b/player/command.c
@@ -295,11 +295,26 @@ static int mp_property_av_speed_correction(void *ctx, struct m_property *prop,
 {
     MPContext *mpctx = ctx;
     char *type = prop->priv;
+    double val = 0;
     switch (type[0]) {
-    case 'a': return m_property_double_ro(action, arg, mpctx->speed_factor_a);
-    case 'v': return m_property_double_ro(action, arg, mpctx->speed_factor_v);
+    case 'a': val = mpctx->speed_factor_a; break;
+    case 'v': val = mpctx->speed_factor_v; break;
+    default: abort();
     }
-    abort();
+
+    if (action == M_PROPERTY_PRINT) {
+        *(char **)arg = talloc_asprintf(NULL, "%+.05f%%", (val - 1) * 100);
+        return M_PROPERTY_OK;
+    }
+
+    return m_property_double_ro(action, arg, val);
+}
+
+static int mp_property_display_sync_active(void *ctx, struct m_property *prop,
+                                           int action, void *arg)
+{
+    MPContext *mpctx = ctx;
+    return m_property_flag_ro(action, arg, mpctx->display_sync_active);
 }
 
 /// filename with path (RO)
@@ -557,6 +572,16 @@ static int mp_property_vo_drop_frame_count(void *ctx, struct m_property *prop,
     return m_property_int_ro(action, arg, vo_get_drop_count(mpctx->video_out));
 }
 
+static int mp_property_vo_missed_frame_count(void *ctx, struct m_property *prop,
+                                             int action, void *arg)
+{
+    MPContext *mpctx = ctx;
+    if (!mpctx->d_video)
+        return M_PROPERTY_UNAVAILABLE;
+
+    return m_property_int_ro(action, arg, vo_get_missed_count(mpctx->video_out));
+}
+
 /// Current position in percent (RW)
 static int mp_property_percent_pos(void *ctx, struct m_property *prop,
                                    int action, void *arg)
@@ -3318,6 +3343,7 @@ static const struct m_property mp_properties[] = {
     {"speed", mp_property_playback_speed},
     {"audio-speed-correction", mp_property_av_speed_correction, .priv = "a"},
     {"video-speed-correction", mp_property_av_speed_correction, .priv = "v"},
+    {"display-sync-active", mp_property_display_sync_active},
     {"filename", mp_property_filename},
     {"stream-open-filename", mp_property_stream_open_filename},
     {"file-size", mp_property_file_size},
@@ -3335,6 +3361,7 @@ static const struct m_property mp_properties[] = {
     {"total-avsync-change", mp_property_total_avsync_change},
     {"drop-frame-count", mp_property_drop_frame_cnt},
     {"vo-drop-frame-count", mp_property_vo_drop_frame_count},
+    {"vo-missed-frame-count", mp_property_vo_missed_frame_count},
     {"percent-pos", mp_property_percent_pos},
     {"time-start", mp_property_time_start},
     {"time-pos", mp_property_time_pos},
@@ -3549,7 +3576,8 @@ static const char *const *const mp_event_property_change[] = {
     E(MPV_EVENT_TICK, "time-pos", "stream-pos", "stream-time-pos", "avsync",
       "percent-pos", "time-remaining", "playtime-remaining", "playback-time",
       "estimated-vf-fps", "drop-frame-count", "vo-drop-frame-count",
-      "total-avsync-change", "audio-speed-correction", "video-speed-correction"),
+      "total-avsync-change", "audio-speed-correction", "video-speed-correction",
+      "vo-missed-frame-count"),
     E(MPV_EVENT_VIDEO_RECONFIG, "video-out-params", "video-params",
       "video-format", "video-codec", "video-bitrate", "dwidth", "dheight",
       "width", "height", "fps", "aspect", "vo-configured", "current-vo",
diff --git a/player/core.h b/player/core.h
@@ -79,6 +79,22 @@ enum seek_precision {
 // Comes from the assumption that some formats round timestamps to ms.
 #define FRAME_DURATION_TOLERANCE 0.0011
 
+enum video_sync {
+    VS_DEFAULT = 0,
+    VS_DISP_RESAMPLE,
+    VS_DISP_RESAMPLE_VDROP,
+    VS_DISP_RESAMPLE_NONE,
+    VS_DISP_VDROP,
+    VS_DISP_NONE,
+    VS_NONE,
+};
+
+#define VS_IS_DISP(x) ((x) == VS_DISP_RESAMPLE ||       \
+                       (x) == VS_DISP_RESAMPLE_VDROP || \
+                       (x) == VS_DISP_RESAMPLE_NONE ||  \
+                       (x) == VS_DISP_VDROP ||          \
+                       (x) == VS_DISP_NONE)
+
 struct track {
     enum stream_type type;
 
@@ -244,7 +260,13 @@ typedef struct MPContext {
     // Redundant values set from opts->playback_speed and speed_factor_*.
     // update_playback_speed() updates them from the other fields.
     double audio_speed, video_speed;
+    bool display_sync_active;
     bool broken_fps_header;
+    double display_sync_frameduration;
+    int display_sync_drift_dir;
+    // Timing error (in seconds) due to rounding on vsync boundaries
+    double display_sync_error;
+    int display_sync_disable_counter;
     /* Set if audio should be timed to start with video frame after seeking,
      * not set when e.g. playing cover art */
     bool sync_audio_to_video;
diff --git a/player/loadfile.c b/player/loadfile.c
@@ -1020,7 +1020,10 @@ static void play_current_file(struct MPContext *mpctx)
     mpctx->max_frames = -1;
     mpctx->video_speed = mpctx->audio_speed = opts->playback_speed;
     mpctx->speed_factor_a = mpctx->speed_factor_v = 1.0;
+    mpctx->display_sync_frameduration = 0.0;
+    mpctx->display_sync_error = 0.0;
     mpctx->broken_fps_header = false;
+    mpctx->display_sync_active = false;
     mpctx->seek = (struct seek_params){ 0 };
 
     reset_playback_state(mpctx);
diff --git a/player/osd.c b/player/osd.c
@@ -225,6 +225,12 @@ static void print_status(struct MPContext *mpctx)
     {
         // VO stats
         if (mpctx->d_video) {
+            if (mpctx->display_sync_active) {
+                saddf(&line, " DS: %f", mpctx->speed_factor_a);
+                int64_t m = vo_get_missed_count(mpctx->video_out);
+                if (m > 0)
+                    saddf(&line, " Missed: %"PRId64, m);
+            }
             int64_t c = vo_get_drop_count(mpctx->video_out);
             if (c > 0 || mpctx->dropped_frames_total > 0) {
                 saddf(&line, " Dropped: %"PRId64, c);
diff --git a/player/video.c b/player/video.c
diff --git a/video/out/vo.c b/video/out/vo.c
diff --git a/video/out/vo.h b/video/out/vo.h
