Keyframes based on factors such as audio level or overall brightness

I got this idea when thinking about making a music video in shotcut where the background would zoom in and out based on audio level. I was wondering if this could be possible in shotcut and it would be really cool to have. You could also have it so audio levels control opacity of layers.
You can have it so factors such as loudness, brightness, amount of a certain colour being shown, amount of certain audio frequency ect
It could maybe be called Factor keyframes and you can set what the lowest keyframes can be and what the highest keyframes can be so for example, if audio levels are higher, it’s closer to the higher keyframes but if it’s silent then it’s closer to the lower keyframe.
It would be great if this idea could be considered.

2 Likes

+1 on this suggestion.
This is a great example of what could happen with the addition of this feature.

I think this proposed name could cause some confusion with the existing keyframes feature. Does anyone have any other suggestions for a name for such an effect?

Other names might be - Audio React Keyframes, Reaction Keyframes, AV Keyframes

Here is a video from FL Studios, and I’m toying with the idea of buying the full package.
It would be my newest play toy. Not sure if Adobe offers such a package.

And this might be one of those to add to the bottom of the roadmap as I could just imagine it’s quite some feat for programming even something simple, compared this. I wanted to suggest for a long time, but didn’t as I didn’t want to be the lone soldier, lol. This style of reaction type editing has captured my attention for many years.

1 Like

Why do you keep putting the word “Keyframes” in the names. That would be inherently confusing. We must be thinking about something different.

I have in mind a new filter that lets you specify some parameters to apply effects to the video (zoom, position, color, etc).

Years ago I create some audio visualization filters for MLT (the video framework underlying Shotcut). Two of them have already been exposed in Shotcut: Waveform and Spectrum. Two others have not been exposed yet: Lightshow and Dance.

The specification for Dance is here:
https://www.mltframework.org/plugins/FilterDance/
There is an example of a video that uses it here:

The specification for Lightshow is here:
https://www.mltframework.org/plugins/FilterLightshow/
There is an example of a video that uses it here:

By the way, the videos in that YouTube channel were all created exclusively with MLT effects.

2 Likes

Because you may not want the same shake effect for a reaction, so it would would work jointly with other keyframes for controlling color, size/position, scale/rotate. You may want the reaction to react for x-frames after an audio level in the 10Hz range, and not react to anything above 160Hz regardless of the level. If someone is playing the cello with a long draw of 5 seconds, you may want the video to slowly slide to the right, then slide back.

I do fear this would add quite a bit of confusion no matter what it’s called and may only be for a small group of folk that may actually want to use it.

I had no idea these existed.
How does one use them with Shotcut?

They are not exposed in the Shotcut GUI yet. It has been on the back of my TODO list to add filter UIs for them. But I wasn’t sure if there would be much demand for them.

I share this concern. Which is why I wouldn’t want to mix it in with the keyframe feature. Have a look at the specification pages for Lightshow and Dance and let me know if they seem useful. I think it would be easier for people to understand (or ignore) a couple of new filters with very specific parameters.

I assume you are already familiar with the “Audio Spectrum Visualization” and “Audio Waveform Visualization” filters that are already available in Shotcut. If not, you should check them out.

1 Like

Oh yes, I love them. I find myself playing with them quite often, trying different things.
rotate%20audio%20spectrum%20curved2b
rotate%20audio%20spectrum%20curved

In fact, there was an issue with someone’s stream, and I mentioned the audio problem. Turns out it was a family member in the next room watching a football game. Depending upon your hearing, you might not hear it. With headphones you can hear it. I made a short video, and only 1/2 of us could hear, even though I isolated the bits showing the audio reaction. The areas highlighted in green has no reaction to keyboard keys or game audio.

1 Like

Maybe reactors or reaction filters.

1 Like

+1 ! :slight_smile:

@brian Dance filter in mlt framework has following parameters :
up, down, left, right, zoom, clockwise, counter clockwise, threshold etc. these parameters allows quick way to bounce in out, shake effect with the audio which I frequently see in music videos. But this effect can be done with beizer keyframe (
linear keyframe can not achive perfect effect eg. bounce ) for few seconds effect. So, Dance filter would be great for applying effect to entire audio duration. So that , in future if beizer keyframe available, this Dance filter still useful.

For audio react anything (eg. brightness,contrast, saturation, etc.) It is technically difficult to address. I have no idea about this. But dance filter’s parameters are enough for audio react effect in my opinion.

After watching videos of this type of effect, I think, “Audio react” video filter would be great.
Thank you.

Made this with conventional filters.

It would be nice if the dance and light show filters would be added sometime in the near future.

Took about a week to make. Put in about an hour a day. The lyrics needed a lot of syncing. I think overall it looks reasonable. The lyrics are a bit explicit.

Muse Psycho

3 Likes

Nice!

I started on them, but then got distracted by other things. I’ll try to dust them off later this month.

It would be nice to have them, but I know it’s not a priority. Almost anything can be done with the current filters and some work.

BTW the latest build is working very well.:slightly_smiling_face:

@sauron here is a very rough draft of the dance filter. Would you be willing to test it for me and give feedback?

dance.zip (2.3 KB)

The file needs to be unzipped in your shotcut installation under qml/filters.

I am looking for feedback on:

  • General usefulness
  • Ranges and limits for the controls. It would be good to limit it to only useful values.
1 Like

Thanks. I’ll test it.
The first thing I noticed is that this filter only works on a playlist item. Does not work when the clip is put on the timeline.

The filter has a lot of parameters and it will take a bit of testing.

Maybe @scellycraftyt, @x3DGime, and @dipensan would like to assist in the test. Since it’s a music based filter maybe @jonray who is a music teacher can join as well.