This isn't entirely on-topic but I've been trying to understand why AI video editing isn't more common, and thought you might know. I've had an idea for a while to make tennis match highlight videos that show every single point of the match. Tennis has a lot of downtime between points (and even more between games and sets). I just want to tell an LLM: here's a two-hour long video of a tennis match. Strip out all the gaps between points. I'm guessing this would a very expensive frame by frame analysis of the video right now and that's why it's not done. Is that right or are there other reasons?
Yeah, it's probably too expensive and complicated to send all the frames to an LLM. I only send about 30 images at 576x324 which is around 15000 tokens a video, and comes to about $0.045 per video. First I save one frame every second and then loop through them comparing each to find the differences, and send just screenshots which have changed significantly, up to a max of 30. Claude only allows 100 images per API call, so it would be a bit fiddly and costly to handle 7000 frames.
Though, now that I'm thinking about it, you could probably do this locally and just look at the part of the image that has the current score, do some local OCR on it to check if the score has changed each frame, if it has, store the timestamp and then use ffmpeg to extract the correct parts. Probably wouldn't need an LLM at all.
As for editing, one thing I do in my videos is audio keywords so my app can do specific things. For example, I can say "AI, mark what I just said as important." Then when it transcribes the audio and the LLM processes it, it will mark that part as a Distinct Moment with a start and end timestamp, a title and description that will show in my app as a clickable link to that part of the video. I'm thinking of adding more commands for more complex editing too.
