Sync Shotcut to Music Beats/Samples Automatically

The_Tedward · July 22, 2019, 3:06pm

Hey guys! Weird question time

Short version: Can I write a Shotcut macro/plugin to cut video based on audio tones or midi so that I can automagically sync it to songs I make in Ableton?

Long version:
So I’d like to be able to make videos a-la Andrew Huang (see this one: https://www.youtube.com/watch?v=5u139At4S7c). Specifically, I want an automated or semi-automated way of syncing the clips of the horse playing the piano [or insert video here] to the music. He’s done most of his work in Ableton, but the fact that he’s running a Mac and I’m running Windows means I can’t handle everything with Ableton’s video processing.

I’m wondering if there’s a workaround something like what Evan Kale did with Sony Vegas a while back – that is , he wrote some custom scripts to handle his video cuts based on audio tones (here’s the video, sorry if you don’t like the presentation…) https://www.youtube.com/watch?v=p53PxLBQFAk It’s pretty trivial to export a MIDI, WAV or MP3 file from Ableton containing the cuts I want.

I’ve programmed in Java/Python/C down to the level of embedded systems before, so I’m not new to that, but I’d really like to spend the least amount of time possible programming and more time…well, making videos.

Anyone care to point me in the right direction on this? (Super thanks for sticking with the long explanation)

sauron · July 22, 2019, 5:40pm

As far as I know you cannot create a macro or plug-in to automatically sync video to music. If you want to sync video to music it needs to be done manually.

Shotcut has 2 filters, Audio Dance Visualization and Audio Light Visualization that work with video and audio. They can probably be used along with extensive manual cutting and syncing to create what you want.

Paul2 · July 22, 2019, 7:38pm

Hi @The_Tedward

Interesting project.
Actually there might be a way, depending in what format Ableton exports the cut “list”.

Since you can program, you may have to make (just once-off) a convertor of sorts to take the Ableton cut list and convert it to a Shotcut mlt project, essentially a xml file.

My initial thoughts as follows:

Export Ableton cut list (hopefully it’s some sort of xml variant).
Run it through a converter, written in either C, Python, Bash, does not really matter.
Get out a mlt file which you load into SC and do further editing & apply effects, export the final video.

Create some simple “projects” in SC and save them, then have a look at the mlt files, they are pretty easy to understand and should not be too difficult to produce from scratch with reference to a specific video file.

Paul2 · July 22, 2019, 11:18pm

There is another way if you don’t mind a bit of manual work.

Using Audacity or any other audio editor/DAW, put the wanted soundtrack on a stereo track, mono also OK.

Now create another stereo (or mono) track.
Into this, you insert audio “blips” into at the right places.
This “blips” can be any audio tone or even white/pink noise.
What I did find is it makes it easier later in SC to make the duration of each “blip”, 80mS (2 frames) if your video rate is 25 fps but up to you.

Once all done, mute the “blips” track and export only the soundtrack.
Now mute the soundtrack and export only the “blips” track.

Open SC and place the wanted video tracks.
Create an audio track and add the exported soundtrack.
Add another audio track and add the “blips” track to it.

As shown below, the video track/s are on top (the blue), the soundtrack on A1 and “blips” on A2:

Now, whenever you see a “blip”, cut the video at the same point.

Once done, mute “blips” track A2 and export.

Certainly not automatic but does make it quicker cutting the video at the correct places
as the “blips” are already inserted at the right points.
Depending on your needs, it may do the trick.

There is a third way where you insert the “blips” at record time (on the camera).
Will post that in a day or two when I have some spare time.
Essentially it works like this:
If you are recording the footage yourself (like shown in the youtube video you linked to), use a camera with stereo external audio input.
On the left channel, you feed the wanted audio from a mic.
On the right channel, a tone burst, controlled via a switch.
This switch you press (to create the tone burst) at the right time.

From there it’s easy as these will then act as markers for your cuts.
If you are keen, I can draw you up a schematic diagram on how to build it.

The_Tedward · July 23, 2019, 11:25am

Thanks for this boss – I was hoping there was a way to find the blips without having to scrub through the waveform. Really appreciate the thoroughness of it though

The_Tedward · July 23, 2019, 11:26am

Thanks, I’ll have a look

Paul2 · July 23, 2019, 3:11pm

No worries, you’re welcome.

What you can also do if recording from scratch, is use the html+js file I created to generate a 1KHz tone for 0.1 sec on your smart phone by pressing on the “TONE” button.

Take the audio output from the headphone socket and feed it to one of the channels of the mic input of your camera, just watch the audio level.
This way, no need to build any tone generating hardware, just an interface cable between phone and camera.

Tested on the following systems: