All the good boys

This is the basic system I use to override video content while keeping consistency. i.e NOT just stlyzing them with a cartoon or painterly effect.

1. Take your video clip and export all the frames in a 512x512 square format. You can see I chose my doggy and it is only 3 or 4 seconds.
2. Look at all the frames and pick the best 4 keyframes. Keyframes should be the first and last frames and a couple of frames where the action starts to change (head turn etc, , mouth open etc).
3. Copy those keyframes into another folder and put them into a grid. I use https://www.codeandweb.com/free-sprite-sheet-packer . Make sure there are no gaps (use 0 pixels in the spacing).
4. In the txt2img tab, copy the grid photo into ControlNet and use HED or Canny, and ask Stable Diffusion to do whatever. I asked for a Zombie Dog, Wolf, Lizard etc.*Addendum... you should put: Light glare on film, Light reflected on film into your negative prompts. This prevents frames from changing colour or brightness usually.
5. When you get a good enough set made, cut up the new grid into 4 photos and paste each over the original frames. I use photoshop. Make sure the filenames of the originals stay the same.
6. Use EBsynth to take your keyframes and stretch them over the whole video. EBsynth is free.
7. Run All. This pukes out a bunch of folders with lots of frames in it. You can take each set of frames and blend them back into clips but the easiest way, if you can, is to click the Export to AE button at the top. It does everything for you!
8. You now have a weird video.

If you have enough Vram you can try a sheet of 16 512x512 images. So 2048x2048 in total. I once pushed it up to 5x5 but my GPU was not happy. I have tried different aspect ratios, different sizes but 512x512 frames do seem to work the best.I'll keep posting my older experiments so you can see the progression/mistakes I made and of course the new ones too. Please have a look through my earlier posts and any tips or ideas do let me know.

​

NEW TIP:

Download the multidiffusion extension. It comes with something else caled TiledVae. Don't use the multidiffusion part but turn on Tiled VAE and set the tile size to be around 1200 to 1600. Now you can do much bigger tile sizes and more frames and not get out of memory errors. TiledVAE swaps time for vRam.

Update. A Youtube tutorial by Digital Magic based in part on my work. Might be of interest.. https://www.youtube.com/watch?v=Adgnk-eKjnU

And the second part of that video... https://www.youtube.com/watch?v=cEnKLyodsWA