- Base feature
- Changelog file
- Image display and description text in README.md
- Unused files
- Support for several large language models
- Async survey running
- Asking for API keys before they are used
- Bugs in survey running
- Bugs in several question types
- Unused files
- Unused package dependencies
- Improvements in async survey running
- Improvements in async survey running
- Improvements in async survey running
- Added logging
- Better handling of async failures
- Fixed bug in survey logic
- Report functionalities are now part of the main package.
- Fixed a bug in the Results.print() function
- The package no longer supports a report extras option.
- Fixed a bug in EndofSurvey
- Question options can now be 1 character long or more (down from 2 characters)
- Fixed a bug where prompts displayed were incorrect (prompts sent were correct)
- Results now provides a
.sql()
method that can be used to explore data in a SQL-like manner. - Results now provides a
.ggplot()
method that can be used to create ggplot2 visualizations. - Agent now admits an optional
name
argument that can be used to identify the Agent.
- Fixed various issues with visualizations. They should now work better.
- The
answer
component of theResults
object is printed in a nicer format.
trait_name
descriptor was not working; it is now fixed.QuestionList
is now working properly again
- The raw model response is now available in the
Results
object, accessed via "raw_model_response" keyword. There is one for each question. The key is the question_name +_raw_response_model
- The
.run(progress_bar = True)
returns a much more informative real-time view of job progress.
- Various fixes and small improvements
-
New documentation: https://docs.expectedparrot.com
-
Progress bar: You can now pass
progress_bar=True
to therun()
method to see a progress bar as your survey is running. Example:
from edsl import Survey
results = Survey.example().run(progress_bar=True)
Job Status
Statistic Value
─────────────────────────────────────────────────────────────────
Elapsed time 1.1 sec.
Total interviews requested 1
Completed interviews 1
Percent complete 100 %
Average time per interview 1.1 sec.
Task remaining 0
Estimated time remaining 0.0 sec.
Model Queues
gpt-4-1106-preview;TPM (k)=1200.0;RPM (k)=8.0
Number question tasks waiting for capacity 0
new token usage
prompt_tokens 0
completion_tokens 0
cost $0.00000
cached token usage
prompt_tokens 104
completion_tokens 35
cost $0.00209
- New language models: We added new models from Anthropic and Databricks. To view a complete list of available models see edsl.enums.LanguageModelType or run:
from edsl import Model
Model.available()
This will return:
['claude-3-haiku-20240307',
'claude-3-opus-20240229',
'claude-3-sonnet-20240229',
'dbrx-instruct',
'gpt-3.5-turbo',
'gpt-4-1106-preview',
'gemini_pro',
'llama-2-13b-chat-hf',
'llama-2-70b-chat-hf',
'mixtral-8x7B-instruct-v0.1']
For instructions on specifying models to use with a survey see new documentation on Language Models. Let us know if there are other models that you would like us to add!
- Cache: We've improved user options for caching LLM calls.
Old method:
Pass a use_cache
boolean parameter to a Model
object to specify whether to access cached results for the model when using it with a survey (i.e., add use_cache=False
to generate new results, as the default value is True).
How it works now:
All results are (still) cached by default. To avoid using a cache (i.e., to generate fresh results), pass an empty Cache
object to the run()
method that will store everything in it. This can be useful if you want to isolate a set of results to share them independently of your other data. Example:
from edsl.data import Cache
c = Cache() # create an empty Cache object
from edsl.questions import QuestionFreeText
results = QuestionFreeText.example().run(cache = c) # pass it to the run method
c # inspect the new data in the cache
We can inspect the contents:
Cache(data = {‘46d1b44cd30e42f0f08faaa7aa461d98’: CacheEntry(model=‘gpt-4-1106-preview’, parameters={‘temperature’: 0.5, ‘max_tokens’: 1000, ‘top_p’: 1, ‘frequency_penalty’: 0, ‘presence_penalty’: 0, ‘logprobs’: False, ‘top_logprobs’: 3}, system_prompt=‘You are answering questions as if you were a human. Do not break character. You are an agent with the following persona:\n{}’, user_prompt=‘You are being asked the following question: How are you?\nReturn a valid JSON formatted like this:\n{“answer”: “<put free text answer here>“}‘, output=’{“id”: “chatcmpl-9CGKXHZPuVcFXJoY7OEOETotJrN4o”, “choices”: [{“finish_reason”: “stop”, “index”: 0, “logprobs”: null, “message”: {“content”: “```json\\n{\\“answer\\“: \\“I\‘m doing well, thank you for asking! How can I assist you today?\\“}\\n```“, “role”: “assistant”, “function_call”: null, “tool_calls”: null}}], “created”: 1712709737, “model”: “gpt-4-1106-preview”, “object”: “chat.completion”, “system_fingerprint”: “fp_d6526cacfe”, “usage”: {“completion_tokens”: 26, “prompt_tokens”: 68, “total_tokens”: 94}}’, iteration=0, timestamp=1712709738)}, immediate_write=True, remote=False)
For more details see new documentation on Caching LLM Calls.
Coming soon: Automatic remote caching options.
- API keys:
You will no longer be prompted to enter your API keys when running a session. We recommend storing your keys in a private
.env
file in order to avoid having to enter them at each session. Alternatively, you can still re-set your keys whenever you run a session. See instructions on setting up an.env
file in our Starter Tutorial.
The Expected Parrot API key is coming soon! It will let you access all models at once and come with automated remote caching of all results. If you would like to test it out, please let us know!
- Prompts: We made it easier to modify the agent and question prompts that are sent to the models. For more details see new documentation on Prompts.
Model
attributeuse_cache
is now deprecated. See details above about how caching now works.
.run(n = ...)
now works and will run your survey with fresh results the specified number of times.
- New models: Run
Model.available()
to see a complete current list.
- A bug in json repair methods.
- A bug in in
Survey.add_rule()
method that caused an additional question to be skipped when used to apply a skip rule.
-
Results
objects now include columns for question components. Call the.columns
method on your results to see a list of all components. Runresults.select("question_type.*", "question_text.*", "question_options.*").print()
to see them. -
Survey
objects now also have a.to_csv()
method.
- Increased the maximum number of multiple choice answer options to 200 (previously 20) to facilitate large codebooks / data labels.
- Methods for setting session caches
New function
set_session_cache
will set the cache for a session:
from edsl import Cache, set_session_cache
set_session_cache(Cache())
The cache can be set to a specific cache object, or it can be set to a dictionary or SQLite3Dict object:
from edsl import Cache, set_session_cache
from edsl.data import SQLiteDict
set_session_cache(Cache(data = SQLiteDict("example.db")))
# or
set_session_cache(Cache(data = {}))
The unset_session_cache
function is used to unset the cache for a session:
from edsl import unset_session_cache
unset_session_cache()
This will unset the cache for the current session, and you will need to pass the cache object to the run method during the session.
Details: https://docs.expectedparrot.com/en/latest/data.html#setting-a-session-cache
-
Answer comments are now a separate component of results The "comment" field that is automatically added to each question (other than free text) is now stored in
Results
ascomment.<question_name>
. Prior to this change, the comment for each question was stored asanswer.<question_name>_comment
, i.e., if you ranresults.columns
the list of columns would includeanswer.<question_name>
andanswer.<question_name>_comment
for each question. With this change, the columns will now beanswer.<question_name>
andcomment.<question_name>_comment
. This change is meant to make it easier to select only the answers, e.g., runningresults.select('answer.*').print()
will no longer also include all the comments, which you may not want to display. (The purpose of the comments field is to allow the model to add any information about its response to a question, which can help avoid problems with JSON formatting when the model does not want to return just the properly formatted response.) -
Exceptions We modified exception messages. If your survey run generates exceptions, run
results.show_exceptions()
to print them in a table.
- A package that was missing for working with Anthropic models.
- New methods for adding, sampling and shuffling
Results
objects:dup_results = results + results
results.shuffle()
results.sample(n=5)
-
Optional parameter
survey.run(cache=False)
if you do not want to access any cached results in running a survey. -
Instructions passed to an agent at creation are now a column of results:
agent_instruction
- New
Survey
method to export a survey to file. Usage:generated_code = survey.code("example.py")
- A bug in
Survey
methodadd_skip_logic()
- Optional parameter in
Results
methodto_list()
to flatten a list of lists (eg, responses toQuestionList
):results.to_list(flatten=True)
- Erroneous error messages about adding rules to a survey.
-
We started a blog! https://blog.expectedparrot.com
-
Agent
/AgentList
methodremove_trait(<trait_key>)
allows you to remove a trait by name. This can be useful for comparing combinations of traits. -
Agent
/AgentList
methodtranslate_traits(<codebook_dict>)
allows you to modify traits based on a codebook passed as dictionary. Example:
agent = Agent(traits = {"age": 45, "hair": 1, "height": 5.5})
agent.translate_traits({"hair": {1:"brown"}})
This will return: Agent(traits = {'age': 10, 'hair': 'brown', 'height': 5.5})
-
AgentList
methodget_codebook(<filename>)
returns the codebook for a CSV file. -
AgentList
methodfrom_csv(<filename>)
loads anAgentList
from a CSV file with the column names astraits
keys. Note that the CSV column names must be valid Python identifiers (e.g.,current_age
and notcurrent age
). -
Results
methodto_scenario_list()
allows you to turn any components of results into a list of scenarios to use with other questions. A default parameterremove_prefixes=True
will remove the results component prefixesagent.
,answer.
,comment.
, etc., so that you don't have to modify placeholder names for the new scenarios. Example: https://docs.expectedparrot.com/en/latest/scenarios.html#turning-results-into-scenarios -
ScenarioList
methodto_agent_list()
converts aScenarioList
into anAgentList
. -
ScenarioList
methodfrom_pdf(<filename>)
allows you to import a PDF and automatically turn the pages into a list of scenarios. Example: https://docs.expectedparrot.com/en/latest/scenarios.html#turning-pdf-pages-into-scenarios -
ScenarioList
methodfrom_csv(<filename>)
allows you to import a CSV and automatically turn the rows into a list of scenarios. -
ScenarioList
methodfrom_pandas(<dataframe>)
allows you to import a pandas dataframe and automatically turn the rows into a list of scenarios. -
Scenario
methodfrom_image(<image_path>)
creates a scenario with a base64 encoding of an image. The scenario is formatted as follows:"file_path": <filname / url>, "encoded_image": <generated_encoding>
Note that you need to use a vision model (e.g.,model = Model('gpt-4o')
) and you do not need to add a{{ placeholder }}
for the scenario (for now--this might change!). Example:
from edsl.questions import QuestionFreeText
from edsl import Scenario, Model
model = Model('gpt-4o')
scenario = Scenario.from_image('general_survey.png') # Image from this notebook: https://docs.expectedparrot.com/en/latest/notebooks/data_labeling_agent.html
# scenario
q = QuestionFreeText(
question_name = "example",
question_text = "What is this image showing?" # We do not need a {{ placeholder }} for this kind of scenario
)
results = q.by(scenario).by(model).run(cache=False)
results.select("example").print(format="rich")
Returns:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ answer ┃
┃ .example ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ This image is a flowchart showing the process of creating and administering a survey for data labeling tasks. │
│ The steps include importing data, creating data labeling tasks as questions about the data, combining the │
│ questions into a survey, inserting the data as scenarios of the questions, and administering the same survey to │
│ all agents. │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
-
Question
andSurvey
methodhtml()
generates an improved html page representation of the object. You can optionally specify the filename and css. See default css:edsl/edsl/surveys/SurveyExportMixin.py
Line 10 in 9d981fa
-
QuestionMultipleChoice
now takes numbers and lists asquestion_options
(e.g.,question_options = [[1,2,3], [4,5,6]]
is allowed). Previously options had to be a list of strings (i.e.,question_options = ['1','2','3']
is still allowed but not required).
-
[In progress] Survey run exceptions are now optionally displayed in an html report.
-
[In progress] New prompt visibility features.
-
[In progress] New methods for piping responses to questions into other questions.
-
[In progress]
QuestionMultipleChoice
is being modified allow non-responsive answers. Previously, an error was thrown if the agent did not select one of the given options. Details TBD.