SDXL, ComfyUI, and Stability AI, where is this heading?

I am truly grateful to Stability AI for providing us with the fantastic foundation model called Stable Diffusion. For that alone, I want to see it stay around for the long term. However, I am not sure if it’s heading in the right direction.

5 months ago, I warned about the sitting water syndrome in my post. And with the introduction of SDXL and the push for Comfy UI, I fear that it is heading in that direction even faster. Instead of complaining about it, I will share my thought on a possible solution in the hope that it may help find the way out.

A node system is what we call a workflow management system. Designing a workflow management system is similar to building a car. You need to have an overall design of the car first and figure out how to fit different components into it. What you can’t do is have a component first and build a car on top of it. It simply doesn’t work that way.

Let me illustrate what I mean. There is something called the base shader in 3D to create material. In Blender, it is called the BSDF shader. I can use it to create every metal imaginable by tweaking the settings. There is no need for a node workflow to do this.


However, when I need to create something more specific such as old worn-out metal with scratches, a node workflow becomes indispensable. Why is that? Unlike a base metal shader which is uniform across all surfaces, things like a worn-out scratched surface aren’t uniform or appear across all surfaces. What this means is that I need to be able to control what shapes, where, and how much of it is going to appear on the surfaces. That is the essence of what a node system allows you to do.



From Houdini to Davinci Resolve Fusion, I can use node workflows to create geometry, material, special effects, motion graphics, and video compositing to name a few. And there is something common in all these node systems. That is looking at every different way a user uses the software, distilling them down to common denominators, and converting those common denominators into nodes. And please don’t tell me AI is any different. I also use chaiNNer which is node based. And it is a pleasure to use because its nodes are the common denominator nodes that allow me to do whatever I want with it.

If you look at the above Blender example, there is a node called the mapping node. It can be mapped from UV maps which are part of the material workflow, object geometry which is a part of 3D modeling, and Camera and Projection which is a part of viewport and object operations. In other words, the mapping node is a common denominator node that can be used to control where things will appear no matter what you need to map to. The same can be said of all the other nodes such as coordinate nodes and math nodes. This allows me to create whatever I need to create in the way I want to create it.

Node workflows can get very complex. Nevertheless, people will embrace complexity if it leads somewhere. But what people don’t want is complexity that leads to nowhere as is the case in SD at the moment. I truly believe it doesn’t have to be this way.

Let me illustrate what I mean. What makes SD special is all the add-ons and extensions the community creates such as ControlNet, Loras, and TI embeddings. I often find myself wondering how wonderful it would be to have a node system to control all these extensions.

For example, Segment Anything, Semantic Segmentation in ControlNet, and Latent Couple all use the same feature, color-coded masks. In fact, from inpainting to face replacement, the usage of masks is prevalent in SD. Yet, there is no mask node as a common denominator node.


For example, the Adetailer extension automatically detects faces, masks them, creates new faces, and scales them to fit the masks. One thing about human faces is that they are all unique. If there is only one face in the scene, there is no need for a node workflow. But if there are several faces, you need a node workflow. If you can imagine a color-coded mask node that can be applied to Adetailer when the faces are being masked, then controlling each face in the scene will become so much easier and simpler.


In my view, things like mask nodes, mapping nodes, and coordinate nodes are essential in the SD workflow. But to do this, it has to start with the perspective of user experience and distill the features into common denominator nodes. At the moment, there are so many extensions that try to do the same thing in so many different ways. If a common denominator node system is there, all these extensions can be mixed and matched to open up all kinds of possibilities for a user to explore and expand. Now that is the kind of complexity I can fully embrace.