V21.04 ALPHA/UNSTABLE - Time Remap

brian · March 7, 2021, 3:56am

Good catch. This is fixed now.

DRM:

7) I’ve experienced the same thing that @MusicalBox is reporting with the Smooth keyframe. The behavior of it has to be looked at because it’s too erratic to the point of being useless as it’s so unreliable. This doesn’t happen on other keyframes with curves. I’m expecting a smoother transition between two speed points but sometimes it just decides to reverse footage and go forward again and there is nothing on the curves to show that it’s doing that. To reproduce this right after adding the Time Remap filter add three keyframes that are about 4 seconds apart from each other. Do not alter the time or speed on any of them. Just add those three keyframes. Then change the second of those three keyframes to smooth. When you play it back and it gets closer to that second keyframe you will see it reverse for no reason even though no change to speed or time was applied. Here is a demo .

The reason that this occurs is because for smooth keyframes Shotcut tries to fit a line for the smooth keyframe, the two keyframes before it, and the two keyframes after it. In your demo, the “2nd keyframe after” is all the way at the end of a long clip. You can “fix” your same example by adding one more keyframe one second after the last keyframe. By doing that, you can “coerce” the line fit to be more subtle and it will not try to go backwards. I agree that this might be unexpected or counter-intuitive for users. I do not like applications where there is a “trick to it”. But it is also a lot of work to implement a new curve fit algorithm. I hope that with this additional explanation some people can experiment further and we can brainstorm ways to make it more intuitive without throwing away what we have. Give it a try, and let me know what you experience.

I experience this sometimes, too. It has not been a priority for me to investigate. I think it might occur more often when the clip sampling frequency does not match the playback sampling frequency (48,000). If you have some time to experiment with sampling frequencies, let me know if you find any correlation.

I have a work-in-progress to improve this. Hopefully it will be available to test in a week. I will be interested in your feedback.

I believe this has been existing behavior for some time and can be seen when using other filters. I do not plan to address it in the next couple of months.

This will be part of the work-in-progress that I have.

Fixed. thanks.

I think this is a good idea in general. I do not plan to do this right away. But we should consider it for an improvement after the time remap has been released.

Great idea. This is done.

I have to admit that the blend mode is very crude. I believe that we occasionally see frames blended when the speed is 1.0000x due to rounding errors. I think that instead of adding an option, I just need to make the blend mode match people’s expectations.

For 1.0x, there should be no blending as you suggest.
For <1.0x, it should blend between two frames proportional to the distance between the two frames of the ideal time. For example, if the mapped time calculates to frame 1.25, then it should blend frame 1 at 75% and frame 2 at 25%.

I will keep this in mind and look for a good opportunity to improve the blending logic.

I hope it doesn’t come to that. Much of the power of the time remap effect is the ability to smoothly change the speed. I think we just need to find good ways to make the keframe UI intuitive for people in these complicated scenarios. Somehow they need to know to add more keyframes to coerce the curve to fit their desires.

kagsundaram · March 7, 2021, 4:36am

This looks like it is based on cubic equations.
(If it is not, ignore all that follows.)

Yes.

But how hard is it in software to tweak the end conditions before applying the algorithm? Can the software add the “one more Keyframe” as a phantom at each end, and then snip it off after the algorithm has done the hard math?

There is still the problem of knowing where to put the phantom point to avoid the undershoot.
Call your last three points Kn-3, Kn-1, Kn, and your phantom point K’.
Looking at how a cubic fits to points, any time you have a sharp descent from Kn-2 to Kn-1, then flat or a shallow descent from Kn-1 to Kn, the cubic will undershoot. However, if the slope and distance of Kn to K’ is made the same as the slope and distance as kn-2 to Kn-1, the point Kn-1 is forced to be the bottom of a curve, and there will be no undershoot between Kn-1 and K’, as you are getting now.
Of course, the phantom point K’ will be deep in undershoot territory, but that is “past the end”, and you are not using that; it only serves to fix the Kn-1 to Kn without changing the algorithm itself.

shotcut · March 7, 2021, 6:15pm

For other filters, the curve is not a big problem as long as the filter correctly handles values out-of-range. We recently found this problem with opacity and fixed it. Time Remap is special because it strongly couples direction with speed making it too easy to get unexpected results than can be difficult to see and understand simply by the curved line. Even our power users here have expressed confusion. Limiting to linear does not break the ability to smoothly change the speed. Humans do not see speed changes easily, and the addition of 1 or 2 linear key frames ought to be good enough. I volunteer to work on the change.

It is a Catmull-Rom spline. Here is where it chooses points:

github.com

mltframework/mlt/blob/master/src/framework/mlt_animation.c#L434


      
                          p[strlen(p) - 1] = '\0';
                          value = &p[1];
                      }
                  }
          
                  // Special case - frame < 0
                  if (item->frame < 0)
                      item->frame += mlt_animation_get_length(self);
          
                  // Set remainder of string as item value.
                  mlt_property_set_string(item->property, value);
                  item->is_key = 1;
              } else {
                  error = 1;
              }
          
              return error;
          }
          
          /** Load an animation item for an absolute position.
           *

and

github.com

mltframework/mlt/blob/master/src/framework/mlt_property.c#L1289


      
                                                      colors[2].g,
                                                      colors[3].g,
                                                      progress),
                              0,
                              255);
              value.b = CLAMP(catmull_rom_interpolate(colors[0].b,
                                                      colors[1].b,
                                                      colors[2].b,
                                                      colors[3].b,
                                                      progress),
                              0,
                              255);
              value.a = CLAMP(catmull_rom_interpolate(colors[0].a,
                                                      colors[1].a,
                                                      colors[2].a,
                                                      colors[3].a,
                                                      progress),
                              0,
                              255);
          } else {
              value = colors[1];

Bear in mind that changing this in any way changes existing behavior of all smooth keyframes of all parameters of all filters of all existing projects. Yes, we sometimes make breaking changes but try not to and sometimes need to make hard decisions. We went through a lot of that in 2020. Now, we already face another difficult one with a non-backward-compatible project file format change with this code branch. The decision to make here is not hard because there is a simple resolution: It must be added as new.

kagsundaram · March 7, 2021, 6:20pm

Agree wholeheartedly.
I recently had a Size, Pozition, Rotate animation PNG overlay that had an abrupt “jerk” between linear Keyframes; I easily fixed it with a few transition linear Keyframes.

shotcut · March 13, 2021, 8:30pm

This thread is closing for the new beta thread; however, I want to mention that I think this is a good idea and the option can be added for Convert in general. I will attempt to add that for the release candidate. Like Reverse, it will also add up to 15 seconds of additional footage before and after to facilitate transitions and revising edits. Currently, when a convert job completes it replaces the source of all matching clips in the project. When this option is turned on, it will not do that and only replace the currently selected one.

shotcut · March 13, 2021, 8:46pm

Please go to the beta version thread now