Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[datasets][gql] addSpansToDataset #3178

Closed
Tracked by #2017
mikeldking opened this issue May 13, 2024 · 2 comments · Fixed by #3205
Closed
Tracked by #2017

[datasets][gql] addSpansToDataset #3178

mikeldking opened this issue May 13, 2024 · 2 comments · Fixed by #3205
Assignees

Comments

@mikeldking
Copy link
Contributor

No description provided.

@axiomofjoy
Copy link
Contributor

axiomofjoy commented May 15, 2024

Proposal

LLM Spans

Copy input messages and output messages as lists of objects. Place them in JSON objects under the keys "input" and "output", respectively.

Example Input:

{
  "input": [
    {
      "message": {
        "content": "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines.",
        "role": "system"
      }
    },
    {
      "message": {
        "content": "Context information is below.\n---------------------\nRanking models are used by search engines to display query results ranked in the order of the highest relevance. These predictions seek to maximize user actions that are then used to evaluate model performance. \n\nThe complexity within a ranking model makes failures challenging to pinpoint as a model’s dimensions expand per recommendation. Notable challenges within ranking models include upstream data quality issues, poor-performing segments, the cold start problem, and more.  \n\n**Use the 'arize-demo-hotel-ranking' model, available in all free accounts, to follow along.** \n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: How do I use the SDK to upload a ranking model?\nAnswer: ",
        "role": "user"
      }
    }
  ]
}

Example Output:

{
  "output": [
    {
      "message": {
        "content": "To use the SDK to upload a ranking model, you can follow the documentation provided by the SDK. The documentation will guide you through the necessary steps to upload the model and integrate it into your system. Make sure to carefully follow the instructions to ensure a successful upload and integration process.",
        "role": "assistant"
      }
    }
  ]
}

Retriever Spans

Copy input and retrieved documents. If the input is plain text, copy it into a JSON object under the key "input".

Example Input

{
    "input": "How do I use the SDK to upload a ranking model?"
}

Example Output

{
  "output": [
    {
      "document": {
        "score": 0.8022561073303223,
        "content": "\nRanking models are used by search engines to display query results ranked in the order of the highest relevance. These predictions seek to maximize user actions that are then used to evaluate model performance. \n\nThe complexity within a ranking model makes failures challenging to pinpoint as a model’s dimensions expand per recommendation. Notable challenges within ranking models include upstream data quality issues, poor-performing segments, the cold start problem, and more.  \n\n\n\n",
        "id": "ad17eeea-e339-4195-991b-8eef54b1db65"
      }
    },
    {
      "document": {
        "score": 0.7964192032814026,
        "content": "\n**Use the 'arize-demo-hotel-ranking' model, available in all free accounts, to follow along.** \n\n",
        "id": "0ce66871-4a50-4d2f-94d2-1531924bf48a"
      }
    }
  ]
}

Other Span Kinds

Copy input and output. If either is plain text, place into a JSON object under the keys "input" and "output", respectively.

@mikeldking
Copy link
Contributor Author

Note you need to copy over prompt variables as well as they function as input

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
2 participants