what about adding blending modes?

[ubaldot]

Description of proposed feature

Given that you touched colours, I have a provoking thought: what about adding blending modes in the future?

At first glance the VMObject class shall be modified such that it takes an additional parameter "blending_mode=" that can take values in an enum {normal (default), add, multiply, screen, ...}.

Blending modes explained: https://www.youtube.com/watch?v=EOoMakEVh_s

[MrDiver]

I mean it general it sounds like a nice idea but i am not sure how well that would go with the mobject system, because normally how the blend mode works is that it Acts on the last layer and the mobject rendering only has an order and not really a notion of layer which would result it somewhat undefined behavior, also i am unsure how well cairo would handle that. If you have a usecase and some more specific ideas on how to implement it we can think about it

[ubaldot]

I think that Manim order and layers can be placed in a 1-1 correspondence, if the set of mobjects used internally in Manim keeps the property of “totally ordered” set.
For example, in the submobjects attribute of Scene the elements are ordered from the bottom to the top, and there I can see layers of course. If the internal machinery of manim does something special that mix up things I dunno of course, I am a user, not a developer. :)

When it turns into blending modes, that is how pixels are interpolated.
If the operation is unary (e.g. ”Take the pure black and make it transparent”) order shall not matter.
Otherwise, if the Commutative Property applies, then order does not matter (for example Multiply: the multiplication is commutative) otherwise it does (e.g. Subtract I think).

A use case? Well, literally every program that deals with computer graphics have blending modes.
The same applies for functions like “bring to front”, “send to back”, etc., related on how mobjects are rendered on the screen.
Regarding the latter, I do that manually now, but note that today these are considered standard features.

[MrDiver]

I would note back that Manim is not really a "graphics" program in that sense because it is not meant for editing images and so on. And we can also have 3D animations in the future which will probably interfere a lot with the idea of blending modes.

So i agree most of the applications that deal with editing images has blending modes, but Manim is a 2D/3D Animation program which does not really go into that use case.
Also with the bring to front and send to back function it would not really make sense in a 3D context. Only 2D and very limited space of animations

Hope that explains the context a little and why i asked about a specific usecase.

[ubaldot]

Nope. Manin is indeed a graphics program for a particular use-case.
Generalizing "adding blending modes" to "editing images and so on" as you did is a mystification. Nobody claimed that Manim is a tool for "editing images".

Only 2D and very limited space of animations.

False. Send to Beck/Bring to front don't represent a limited space of usage in 2D context. They are widely used in 2D context.

I also find it interesting that you close an opinionated issue on behalf of the whole community.
What if someone wants to implement this feature ?

[behackl]

I've moved this from issues (which should, ideally, represent more fleshed out and concrete tasks that can be tackled) to a discussion. There are some things that are not clear to me yet.

It would be very interesting to construct a more concrete use case, specifically in the context of a task you would typically use Manim for. What sort of Mobject construction or Animation would benefit from having blending modes available?

I know that many other graphics programs have blending modes (and actually also our current rendering backend, cairo, supports something like that). While I can see how these are useful in particular in the context of users creating static content using a GUI, I am not sure how useful they are in a mainly programmatic context. I am sure it is possible to come up with a good scenario where having blending modes is useful. And ideas?

Leaving the use case aside, let's briefly think about the implementation. I can technically see, perhaps, how color blending might work in the context of 2D mobjects as, indeed, there is a global mobject order which can be seen to be equivalent to the concept of layers. The implementation would probably still be somewhat complicated, but I could see it happening. The equivalency to layers breaks down though (I think?) as soon as we move to 3D scenes, where it is not so clear whether it makes sense to attach a blending mode attribute to singular mobjects.

I can see them being useful in a context like https://developers.arcgis.com/javascript/latest/sample-code/scene-blendmode/ -- but I am not sure this is an example that can be transferred to context that is particularly meaningful for us.

[ubaldot]

Regarding the use-case: I think there are plenty of use case, on top of my head I am thinking if you want to overlap something on level-surfaces, to show how quantization work and such, but also for aesthetic which IMHO is extremely important.

Take for example these two examples (it is a frame Manim - Final Cut Pro).
One is with blending mode Multiply the other with Normal. I think the difference is quite huge.

Regarding the implementation: I don't know, this should be a dev discussion I think.

Regarding the send to back/bring to front: can't this feature be only available in 2D and disabled in 3D? Otherwise, in 3D the first thing that comes in my mind (it may all wrong, I am just writing it as it is coming in my mind) couldn't be to consider the normal vector coming out from the screen of your computer as layering reference? And then to consider send to back/bring to front wrt to such a normal vector?

[behackl]

on top of my head I am thinking if you want to overlap something on level-surfaces, to show how quantization work and such, but also for aesthetic which IMHO is extremely important

I don't have a good idea what exactly you mean here, but the other concrete example is a nice usecase.

We can't fully decouple the discussion around a potential implementation; the decision whether this is a feature that we want to have also depends on what the architecture can support. I am not sure how things would work in an OpenGL setting (@MrDiver might have a much better idea), but given our Mobject-based design it is not something that we can just slap in.

Regarding the send to back/bring to front

Everything in Manim is in principle in 3D, we currently don't really have a clean distinction between 2d and 3d scenes. That's also why things are not as simple ™️

We are working on options that allow more control over how things are ordered in this 3D space, there are some fun examples in our dev channels on Discord. :-)

[MrDiver]

So one thing i already had in mind for experimental, which is not in the scope of the near future but it is a compositor like node structure, which can be used to combine different scenes and playbacks onto each other, but other than that it just poses a LOT of problems especially with the 3D application part.

And i would not like to complicate the architecture more with separating 2D and 3D again it will probably unnecessary balast in the long run.

And yes the aesthetic part is a thing but there are also other ways to achieve a similar thing, so for the current main branch I wouldn't really suggest this feature as it introduces more problems than it solves.

@behackl for the OpenGL part I wouldn't necessarily say it's a problem for the implementation but more the usefulness.

If there is a consensus on a good way to handle blending modes in 3D and layering then i will be Happy to implement it

And also again for the functions with the ordering, the mobjects will not be ordered in the future and just rendered by depth so the only thing you can do is deactivating the automatic depth testing and then change the order manually if you want a specific order to render things.
And implementing something like a bring to front function is not really feasible in 3D because it could mean anything and i do not like to make assumptions for the users.

[ubaldot]

Fine. I thought that adding blending modes along with send to back/bring to front was a sort of "low hanging fruit" but clearly is not due to that everything is 3D. For me we can drop the idea.

are also other ways to achieve a similar thing,

You mean through external programs? If so, yes, I agree.

[MrDiver]

Nope. Manin is indeed a graphics program for a particular use-case.
Generalizing "adding blending modes" to "editing images and so on" as you did is a mystification. Nobody claimed that Manim is a tool for "editing images".

And also xD I still have to disagree. So either give a clear definition on what a "graphics" program is, otherwise I will give you a ton of examples where this is not an existing functionality. Because arguably "graphics" programs are kind of all graphical applications if you claim Manim is one of them.

So let's look at games, I do not really see any single game that includes blending modes for any reason, and those are pretty graphic I think.

Then your argument becomes: No one claimed Manim is a game

So sure let's go to the closest thing to Manim that exists in the open-source World which would be Blender

• https://blender.stackexchange.com/questions/82267/blending-modes-in-blender
• https://docs.blender.org/manual/en/2.79/render/blender_render/textures/properties/influence/blending_modes.html

So please tell me if your definition of graphical program suits a program like Blender, because the general pipeline is basically what Manim does.
And now if Blender suits your definition of a graphical program, then also please tell me why does blender not have Blending modes for 3D objects and only supports it for textures which was also kindly deleted after version 2.8.

Right it's because it is not just about being a "graphical" program.

Because as per definition in the code Manim is a 3D Animation program which has a procedural way of creating Animations through code. Which is a "graphical" application but to say that every graphical application has blending modes is a pretty stubborn viewpoint.

[MrDiver]

Fine. I thought that adding blending modes along with send to back/bring to front was a sort of "low hanging fruit" but clearly is not due to that everything is 3D. For me we can drop the idea.

Maybe understand the full scope of the problem first and help with finding a solution, and provide use cases for ideas, for example "I want to do this specific animation, but it is not possible because xyz is currently not possible in manim". Which would give a starting point for a problem.

Which will be way more constructive in actually finding a solution to implement the feature that you want, and then help to find a solution if we state that it will be more difficult than you first had in mind.

And as a last note, please do not use the most general category to describe a software and use arguments like "every other software has this" it will never result in a constructive discussion, because Manim is not every other software.

[MrDiver]

So if you find a good use case and provide maybe an example how to solve this/implement it in Manim then I reiterate I would be happy to implement it!

[ubaldot]

With this I am going to close.

I already provided a use case.

Anyhow, @behackl provided a definitive answer here:

Everything in Manim is in principle in 3D, we currently don't really have a clean distinction between 2d and 3d scenes. That's also why things are not as simple ™️

Knowing that in Manim there is no distinction between 2D and 3D answers all the questions. It was enough to say it at the beginning. From the provided answers one can only understand that is a 2D/3D program, not that there is no distinction in the internal machinery.

xD

Excuse me‚ why you need to Laugh Out Loud? Is it normal that a dev laugh out loud to a user?

[behackl]

Knowing that in Manim there is no distinction between 2D and 3D answers all the questions. It was enough to say it at the beginning. From the provided answers one can only understand that is a 2D/3D program, not that there is no distinction in the internal machinery.

Well, yes and no. From your point of view it perhaps answers the question whether or not this is a low-hanging fruit (it unfortunately is not) -- but in particular in the context of "flat" mobjects, this is still a feature that might be interesting, situationally. You might have run into the fact that there is no distinction between 2D and 3D when using Manim already due to the fact that all of our directions and points are always given as vectors with three components.

The reason for us trying to nail down specific usecases is because there might be ways to allow an implementation not in general for all mobjects, but perhaps for some specialized ones. There could be a global setting for the camera, say, or an implementation for groups of "flat" mobjects. Figuring out which of these technical approaches might suit users best requires the collection of as-specific-as-possible use cases. Your example gives a good idea of what you'd like to achieve, it is not quite specific enough (Which Mobjects are present in your scene here? Is the blending mode set on some Group, do we need a special mobject representing the background?) for us to reach a conclusion.

Excuse me‚ why you need to Laugh Out Loud? Is it normal that a dev laugh out loud to a user?

It is hard to convey intentions behind written text and emoji in a medium like this. Knowing MrDiver, his "xD" was not meant to attack or offend you personally in any way. I don't think it is fruitful to discuss the tone of the discussion here any further, that usually does not help with anything. Come join us in our discord, we're in the voice channels there quite frequently, and let's chat about this discussion, if you like. I do not want anyone going out of this angrily or frustrated.

I'll definitely leave this dicussion open; the idea is interesting and I think it might be worth considering some sort of partial solution for "flat" mobjects; perhaps it fits in some way to the different render orderings for groups that we have been working on. It surely does not do any harm to keep the idea around.

[ubaldot]

@behackl I appreciate your effort to smooth things down and your interest in the topics I raised. However, I finally get clear answers and I don't feel we need further clarification on this topic. But if you really want to talk to me face to face then fine, ping me on discord (maybe not tonight because it is Saturday and I am going out).

Having said that, I will gladly use Final Cut Pro for blending modes and Doodly for Whiteboard animations. You follow your roadmap. I will use Manim for what it has to offer. I may change tool if something that better fits my needs show up along the way but at the moment I don't see a valid alternative and I spent too much time in learning Manim so I will keep in using it but certainly without the same expectations as before. Sorry, but I am too old for going through all of this. :D

P.S.
Once more I appreciate your effort in smoothing out things, but unfortunately the claim that "xD" may be misinterpreted falls short given that the author confirmed his/her intention in the issue related to the whiteboard animation. Devs that laugh at users is a violation of the code-of-conduct but tbh I don't expect any action on that, nor I expect that it won't happen anymore in the future, but mind that would be very bad for the tool reputation in the long run IMHO.

[behackl]

and I don't feel we need further clarification on this topic

Sure! We'll keep collecting more concrete use cases, and maybe we can implement some partial solution at some point.

You follow your roadmap. I will use Manim [...] but certainly without the same expectations as before

Not exactly sure what you mean here, but this seems like a healthy approach. We only have a very small team of active developers, and most resources are currently allocated to refactoring the internals of the library.

violation of the code-of-conduct

This is important to me, which is why I don't want to leave it at that: I read MrDiver's message differently than you; I've seen it before I replied here. I do not come to the same conclusion that the emoji was meant to attack you personally. I rather read it as an expression of him being frustrated with the discussion, not as an act of belittling you. You've met MrDiver in our voice channel already, from what I could gather this was not an unpleasant conversation. While the two of you might not share the same opinion, I am fairly sure this is more of a misunderstanding rather than malicious behavior on either side. Do reach out to me directly (or just join us when you catch us in one of our voice channels) in case you would like to discuss this further. I do take this very serious.

[ubaldot]

You've met MrDiver in our voice channel already, from what I could gather this was not an unpleasant conversation.

Yep, I just passed by the chat during my "discord how does it work?" self-discovering phase.

Do reach out to me directly (or just join us when you catch us in one of our voice channels) in case you would like to discuss this further

Sure thing. I actually pinged you few days ago. I will try again next time I will see you online (or please you do it when you see me online).

[behackl]

Did you ping me somewhere on our server? I usually see those, sorry. I won't be available for chatting this weekend, but might hang out in our voice channels next week.

[ubaldot]

Yep. I'll try to ping you right now. Let me know if you get something (it may just be myself who cannot really use discord).

[jsonvillanueva]

I think for the sake of Cairo at least, the changes sound possible. They'd likely happen somewhere in the camera.py between capture_mobjects => display_func => display_multiple_vectorized_mobject
and the various other subcalls:

manim/manim/camera/camera.py

Lines 633 to 756 in 4e3cfd2

	def display_multiple_non_background_colored_vmobjects(
	self, vmobjects: list, pixel_array: np.ndarray
	):
	"""Displays multiple VMobjects in the cairo context, as long as they don't have
	background colors.
	Parameters
	----------
	vmobjects
	list of the VMobjects
	pixel_array
	The Pixel array to add the VMobjects to.
	"""
	ctx = self.get_cairo_context(pixel_array)
	for vmobject in vmobjects:
	self.display_vectorized(vmobject, ctx)
	def display_vectorized(self, vmobject: VMobject, ctx: cairo.Context):
	"""Displays a VMobject in the cairo context
	Parameters
	----------
	vmobject
	The Vectorized Mobject to display
	ctx
	The cairo context to use.
	Returns
	-------
	Camera
	The camera object
	"""
	self.set_cairo_context_path(ctx, vmobject)
	self.apply_stroke(ctx, vmobject, background=True)
	self.apply_fill(ctx, vmobject)
	self.apply_stroke(ctx, vmobject)
	return self
	def set_cairo_context_path(self, ctx: cairo.Context, vmobject: VMobject):
	"""Sets a path for the cairo context with the vmobject passed
	Parameters
	----------
	ctx
	The cairo context
	vmobject
	The VMobject
	Returns
	-------
	Camera
	Camera object after setting cairo_context_path
	"""
	points = self.transform_points_pre_display(vmobject, vmobject.points)
	# TODO, shouldn't this be handled in transform_points_pre_display?
	# points = points - self.get_frame_center()
	if len(points) == 0:
	return
	ctx.new_path()
	subpaths = vmobject.gen_subpaths_from_points_2d(points)
	for subpath in subpaths:
	quads = vmobject.gen_cubic_bezier_tuples_from_points(subpath)
	ctx.new_sub_path()
	start = subpath[0]
	ctx.move_to(*start[:2])
	for _p0, p1, p2, p3 in quads:
	ctx.curve_to(*p1[:2], *p2[:2], *p3[:2])
	if vmobject.consider_points_equals_2d(subpath[0], subpath[-1]):
	ctx.close_path()
	return self
	def set_cairo_context_color(
	self, ctx: cairo.Context, rgbas: np.ndarray, vmobject: VMobject
	):
	"""Sets the color of the cairo context
	Parameters
	----------
	ctx
	The cairo context
	rgbas
	The RGBA array with which to color the context.
	vmobject
	The VMobject with which to set the color.
	Returns
	-------
	Camera
	The camera object
	"""
	if len(rgbas) == 1:
	# Use reversed rgb because cairo surface is
	# encodes it in reverse order
	ctx.set_source_rgba(*rgbas[0][2::-1], rgbas[0][3])
	else:
	points = vmobject.get_gradient_start_and_end_points()
	points = self.transform_points_pre_display(vmobject, points)
	pat = cairo.LinearGradient(*it.chain(*(point[:2] for point in points)))
	step = 1.0 / (len(rgbas) - 1)
	offsets = np.arange(0, 1 + step, step)
	for rgba, offset in zip(rgbas, offsets):
	pat.add_color_stop_rgba(offset, *rgba[2::-1], rgba[3])
	ctx.set_source(pat)
	return self
	def apply_fill(self, ctx: cairo.Context, vmobject: VMobject):
	"""Fills the cairo context
	Parameters
	----------
	ctx
	The cairo context
	vmobject
	The VMobject
	Returns
	-------
	Camera
	The camera object.
	"""
	self.set_cairo_context_color(ctx, self.get_fill_rgbas(vmobject), vmobject)
	ctx.fill_preserve()
	return self