Break apart session processor and the running of each session into se…

[brandonrising]

…parate classes

Summary

Breaks up the currently scary to update session processor into separate components and provides callback function passthroughs which allow you to define tasks be run before/after sessions, as well as before/after each node.

QA Instructions

Run invocations of various types and batch sizes. Compare output images of same inputs against those generated on current main. Validate logging/stats are formatted correctly.
Intentionally run malformed graphs and nodes which will error out to ensure error reporting is still correct.

Merge Plan

I will merge once the PR has been thoroughly reviewed, tested, and approved on multiple OS/environments.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Tests added / updated (if applicable)
• Documentation added / updated (if applicable)

[lstein]

@brandonrising Do you mind taking a look at the multi-GPU support PR #5997 to estimate how difficult it would be to integrate that into what is done here? That PR has done a small amount of refactoring to break the big scary loop into smaller pieces.

[psychedelicious]

Notes from my changes:

• Rearranged the session callbacks to be in the session runner. IMO this makes sense, all the session and “below” logic is in the session runner. All session callbacks are provided on init to the runner.
• Added a node error callback to session runner.
• Added a nonfatal processor error callback, provided on init to the processor.
• Consolidated some of the callback logic. The main loops are now very clean.
• Made the callbacks lists and cleaned up a some naming. This lets us better separate concerns when we want to do multiple things during the queue item lifecycle - we don’t have to smoosh all logic in a single callback per injection point.
• Fixed an issue that would prevent the profiler from ever profiling anything.
• Created protocols for each callback. This lets them take kwargs instead of positional args, which is a bit safer.
• Some test callbacks are in dependencies.py for smoke testing

I resolved a longstanding issue where pydantic validation errors for nodes were handled as processor-level errors, instead of node-level errors.

@brandonrising had almost fixed it in the first iteration of this PR, but errors were being set on the previous node, when they should be on the current node.

To resolve this, some error handling was added to the session.next() method that prepares the nodes. This allows us to get the node in its state just before it failed validation and mark the current node as failed. An new NodeInputError represents this failure.

This makes our error reporting more accurate. It also is a better UX - the error messages are very clear.

[psychedelicious]

Here's how I smoke tested:

• Do normal generations, single and multiple iterations
• Cancel individual generations and the whole queue
• Create a workflow with a single divide integer node, leave it at defaults and invoke. Confirm the error is handled as expected.
• Create a workflow with a single integer primitive @ 0, connected to the width of a resize image node. This creates a NodeInputError. It should be handled as expected.
• Set profile_graphs: true in the config. It should write out the profiles.

[psychedelicious]

I've updated a few things in this PR:

• error_type, error_message and error_traceback are new queue item attributes. They are added via DB migration.
• Add handling for these new error attributes in the processor
• Use the new error message attributes in the error toast
• Trigger error toast on queue item failure instead of invocation error, this lets us surface errors that occurred outside the node
• Fix a race condition where nodes were not marked as having an error (this was introduced earlier in this PR by me)

Working on this has underscored the need for a way to test the processor (and new session runner). I'm not familiar enough with mocking to have a clear idea of how we can do this, but I think it's feasible - especially now that things are modularized.