-
Notifications
You must be signed in to change notification settings - Fork 43k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weβll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(agent): Introduce Python code execution as prompt strategy #7142
base: master
Are you sure you want to change the base?
Conversation
β Deploy Preview for auto-gpt-docs canceled.
|
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7142 +/- ##
==========================================
+ Coverage 35.01% 42.92% +7.91%
==========================================
Files 18 83 +65
Lines 1211 4883 +3672
Branches 179 675 +496
==========================================
+ Hits 424 2096 +1672
- Misses 758 2691 +1933
- Partials 29 96 +67
Flags with carried forward coverage won't be shown. Click here to find out more. β View full report in Codecov by Sentry. |
Should this be in forge?? |
CI Failure Feedback π§(Checks updated until commit ca7ca22)
β¨ CI feedback usage guide:The CI feedback tool (
In addition to being automatically triggered, the tool can also be invoked manually by commenting on a PR:
where Configuration options
See more information about the |
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
from .base import BaseAgent, BaseAgentActionProposal | ||
from .prompt_strategies.one_shot import OneShotAgentActionProposal |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OneShotAgentActionProposal is not part of .agent
@@ -97,19 +103,20 @@ def __init__( | |||
llm_provider: ChatModelProvider, | |||
file_storage: FileStorage, | |||
legacy_config: Config, | |||
prompt_strategy_class: type[PromptStrategy] = CodeFlowAgentPromptStrategy, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
decouple agent.py & OneShot. pass this class for custom prompt strategy
if not parsed_response.python_code: | ||
raise ValueError("python_code is empty") | ||
|
||
available_functions = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is for the generated code validation.
autogpts/autogpt/autogpt/app/cli.py
Outdated
@@ -2,18 +2,21 @@ | |||
from logging import _nameToLevel as logLevelMap | |||
from pathlib import Path | |||
from typing import Optional | |||
from dotenv import load_dotenv |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ignore this, for debugging purpose, will revert
logger = logging.getLogger(__name__) | ||
|
||
|
||
class CodeFlowExecutionComponent(CommandProvider): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So... BaseAgentActionProposal use use_tool
which only executes a single tool, and it's already intertwined with the agent implementation. So my approach on having this code flow execution is by returning BaseAgentActionProposal
that use this newly created tool, which is basically executing python code using already existing other tools as variables.
@@ -158,6 +159,37 @@ def fmt_line(self) -> str: | |||
) | |||
return f"{self.name}: {self.description}. Params: ({params})" | |||
|
|||
def fmt_header(self, callable=None) -> str: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
callable part here is unused, will revert
@@ -443,6 +443,8 @@ async def create_chat_completion( | |||
if not parse_errors: | |||
try: | |||
parsed_result = completion_parser(assistant_msg) | |||
if isinstance(parsed_result, Coroutine): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So completion_parser
needs to be async for the code validation, I did this instead of changing the whole usage of it to async.
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
Conflicts have been resolved! π A maintainer will review the pull request shortly. |
This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok in general but I think it's unclear what this does (please update the description). This introduces new prompt strategy and new CommandProvider
component. Also missing newlines at the end in multiple files.
@@ -170,6 +174,7 @@ async def propose_action(self) -> OneShotAgentActionProposal: | |||
# Get commands | |||
self.commands = await self.run_pipeline(CommandProvider.get_commands) | |||
self._remove_disabled_commands() | |||
self.code_flow_executor.set_available_functions(self.commands) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This approach requires the user to explicitly pass commands.
I think it's better to take commands from the agent as opposed to the current to the component. You could for example pass Agent
(or some getter) to the component __init__
and then just access commands
when required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice to see somebody making a component! :)
ruff = "^0.4.4" | ||
pyright = "^1.1.362" | ||
ptyprocess = "^0.7.0" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
unsorted, should be alphabetical
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- imo files in
forge/utils/function
deserve their own forge-level directory, sayforge/code_execution
. - Why not just executing code directly, what this all do?
@property | ||
def return_type(self) -> type: | ||
type = inspect.signature(self.method).return_annotation | ||
if type == inspect.Signature.empty: | ||
return None | ||
return type.__name__ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This declares return as type
but returns str
"""A component that provides commands to execute code flow.""" | ||
|
||
def __init__(self): | ||
self._enabled = True |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Components are enabled by default
def set_available_functions(self, functions: list[Command]): | ||
self.available_functions = { | ||
name: function for function in functions for name in function.names | ||
} | ||
|
||
def get_commands(self) -> Iterator[Command]: | ||
yield self.execute_code_flow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This includes execute_code_flow
in self.available_functions
, is that intended?
- Commands are taken from all
CommandsProviders
(includingexecute_code_flow
) - You pass all commands from agents to the
self.available_functions
Just removeexecute_code_flow
fromself.available_functions
.
# "Your decisions must always be made independently without seeking " | ||
# "user assistance. Play to your strengths as an LLM and pursue " | ||
# "simple strategies with no legal complications.", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think remove if unused
Background
Add support for executing agent steps by running Python code. This allows for more complicated composite function calls within each step of agent execution.
Related:
Changes ποΈ
PromptStrategy
:CodeFlowAgentPromptStrategy
undercode_flow.py
utils.function
package that handle a python code validation that utilizeruff
&pyRight
as the validation engines.CodeFlowAgentPromptStrategy
as the existing default strategy.PR Quality Scorecard β¨
+2 pts
+5 pts
+5 pts
+5 pts
-4 pts
+4 pts
+5 pts
-5 pts
agbenchmark
to verify that these changes do not regress performance? β+10 pts