Last week’s hottest news other than the drama unfolding at Open.AI with Sam, this was a very positive news when Stability.AI announced that they have released Stable Video Diffusion<\/a> (21st November 2023). A few days later the first two models are available for experimental\/evaluation use (ie. Non-commercial), so of course then we wait for the community to take this to next level and make it usable on your own PC.<\/p>\n\n\n\n
We have two models available:<\/p>\n\n\n\n
Here are some samples available and shared at the above links. These are smooth GIF images.<\/p>\n\n\n\n
Now that you have the links to the models you can download the model file (.safetensors) which is about ~10 GB, depending upon what you prefer to use. I will be using Stable Video Diffusion XT version to push my RTX4080 16GB card<\/a>.<\/p>\n\n\n\n
Next we need the ComfyUI workflow which is available for download<\/a>. There is one workflow for Text-to-Image-to-Video and another for Image-to-Video.<\/p>\n\n\n\n
I am going to experiment with Image-to-Video which I am further modifying to produce MP4 videos or GIF images using the Video Combine node included in ComfyUI-VideoHelperSuite<\/a><\/p>\n\n\n\n
The first experiment is using a frame that I have a frame from another project I’ve been working on. The result was pretty cool using the default settings of the workflow. The camera movement is pretty cool as the subject moves the camera is following her.<\/p>\n\n\n\n
This result was produced in 52 seconds in my setup.<\/p>\n\n\n\n
Now when I’m experimenting further with 24fps and producing 2-3 seconds worth of video it takes 2-3 minutes. This is pretty acceptable at 1024 by 576 pixels, 48 video frames, 24 fps – this gives me 3 second clip.<\/p>\n\n\n\n
The key parameters of SVD_img2vid_Conditioning are:<\/p>\n\n\n\n
Tip:<\/strong> If you get this error (below) when running the workflow simply uninstall or disable FreeU.<\/p>\n\n\n\n
Error occurred when executing KSampler:\nunsupported operand type(s) for *=: 'int' and 'NoneType'<\/code><\/pre>\n\n\n\nAll my experiments have been interesting and some produced good results but nothing like the preview videos shared by Stability.AI above. I guess their secret recipe is not shared yet but it would be wonderful to know the settings they were using so we can also replicate them.<\/p>\n\n\n\n\n\n<\/video><\/figure>\n<\/div>\n\n\n\n\n<\/video><\/figure>\n<\/div>\n<\/div>\n\n\n\nI’m continuing my experimentation and particularly with motion_bucket_id to see how it affects the animation. I will share my findings in up coming posts.<\/p>\n","protected":false},"excerpt":{"rendered":"Last week’s hottest news other than the drama unfolding at Open.AI with Sam, this was a very positive news when Stability.AI announced that they have released Stable Video Diffusion (21st November 2023). A few days later the first two models are available for experimental\/evaluation use (ie. Non-commercial), so of course then we wait for the […]<\/p>\n","protected":false},"author":1,"featured_media":10838,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"off","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[1699],"tags":[],"class_list":["post-10827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comfyui"],"_links":{"self":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/comments?post=10827"}],"version-history":[{"count":10,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions"}],"predecessor-version":[{"id":10856,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions\/10856"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media\/10838"}],"wp:attachment":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media?parent=10827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/categories?post=10827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/tags?post=10827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
All my experiments have been interesting and some produced good results but nothing like the preview videos shared by Stability.AI above. I guess their secret recipe is not shared yet but it would be wonderful to know the settings they were using so we can also replicate them.<\/p>\n\n\n\n
I’m continuing my experimentation and particularly with motion_bucket_id to see how it affects the animation. I will share my findings in up coming posts.<\/p>\n","protected":false},"excerpt":{"rendered":"
Last week’s hottest news other than the drama unfolding at Open.AI with Sam, this was a very positive news when Stability.AI announced that they have released Stable Video Diffusion (21st November 2023). A few days later the first two models are available for experimental\/evaluation use (ie. Non-commercial), so of course then we wait for the […]<\/p>\n","protected":false},"author":1,"featured_media":10838,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"off","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[1699],"tags":[],"class_list":["post-10827","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-comfyui"],"_links":{"self":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/comments?post=10827"}],"version-history":[{"count":10,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions"}],"predecessor-version":[{"id":10856,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/posts\/10827\/revisions\/10856"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media\/10838"}],"wp:attachment":[{"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/media?parent=10827"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/categories?post=10827"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/weirdwonderfulai.art\/wp-json\/wp\/v2\/tags?post=10827"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
![\"\"](\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/svd-sample.gif\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/weirdwonderfulai.art\/wp-content\/uploads\/2023\/11\/SVD-Workflow-sample-1024x410.jpg\")