Skip to content

Equistamp 0.0.1

3rd Party AI Evaluation Service Setting & Protecting the Global Standard of AI Safety

Endpoints

GET /auth

Response 200 OK 

{
    "id": "4f7d1940-3db2-464e-9676-a90ea6e0e258",
    "user_name": "string",
    "full_name": "string",
    "user_image": "string",
    "bio": "string",
    "display_options": null,
    "join_date": "2022-04-13",
    "email_address": "string",
    "subscription_level": "string",
    "alarms": [
        {
            "id": "5a011cf1-75b1-42da-9ca3-36baae8febf4",
            "name": "They are coming!!",
            "description": "string",
            "metric": "string",
            "threshold": 10.12,
            "public": true,
            "predicted_trigger_date": "2022-04-13",
            "last_trigger_date": "2022-04-13",
            "trigger_cadence": "string",
            "predicted_line_of_best_fit": null,
            "owner_id": "28ddcbd4-d676-41e6-b401-98a833a72743",
            "evaluation_id": "3f4846df-53f2-465b-8d06-5acd7a48c6c6",
            "owner": "49167a20-69a4-402c-94cc-735a4aba3a55",
            "evaluation": "396213fd-69e2-4045-90bd-533f7c090e15",
            "models": [
                "7f37e840-15e6-4e0e-923e-44b0b3610c0c"
            ]
        }
    ]
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "user_name": {
            "type": "string"
        },
        "full_name": {
            "type": "string",
            "nullable": true
        },
        "user_image": {
            "type": "string",
            "nullable": true
        },
        "bio": {
            "type": "string",
            "nullable": true
        },
        "display_options": {},
        "join_date": {
            "type": "string",
            "format": "date"
        },
        "email_address": {
            "type": "string"
        },
        "subscription_level": {
            "type": "string"
        },
        "alarms": {
            "type": "array",
            "items": {
                "$ref": "#/components/schemas/ShallowAlarm"
            }
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /auth

Log in the provided user, or send an email with a login link.

Description

This endpoint handles logging in, both when valid credentials are provided, and when the user needs to reset their password. This happens depending on the provided JSON body:

  1. If login credentials are provided, then try to log the user in - if this fails, a 401 will be returned
  2. If reset_email is provided, assume that the user has forgotten their password. If this email can be found in the system, then send them an email with a log in link. Either way, this will always return a 200, to avoid leaking email addresses.

Log in credentials are a user identifier and a password. The following are supported:

  • username - this is the user name of the user (not the display name)
  • email - the email of the user
  • login - this will accept either the email or username

The result of logging in is a JSON object with a Session-Token. This should be provided as the Session-Token header on subsequent calls to the API to authenticate the user. The token will expire after a week of inactivity, but otherwise will be refreshed while using the system.

Request body

{
    "username": "mr_blobby",
    "email": "mr_blobby@bla.com",
    "login": "mr_blobby@bla.com",
    "password": "hunter2",
    "reset_email": "bla@bla.com"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "username": {
            "type": "string",
            "example": "mr_blobby"
        },
        "email": {
            "type": "string",
            "example": "mr_blobby@bla.com"
        },
        "login": {
            "type": "string",
            "example": "mr_blobby@bla.com"
        },
        "password": {
            "type": "string",
            "format": "password",
            "example": "hunter2"
        },
        "reset_email": {
            "type": "string",
            "format": "email",
            "example": "bla@bla.com",
            "description": "Used when resetting a password. A login link will be sent to this email, but only if can be found in the system. When missing, this will fail silently, i.e. a 200 will be returned"
        }
    }
}

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "type": "object",
            "description": "Returned when the user successfully logs in",
            "properties": {
                "session_token": {
                    "type": "string",
                    "format": "uuid",
                    "description": "The session token of the logged in user. This should be sent as the \"Session-Token\" header on all subsequent calls. "
                },
                "token_expiration": {
                    "type": "number",
                    "format": "int32",
                    "description": "The POSIX timestamp when this token will expire. Generally in a weeks time."
                }
            }
        },
        {
            "type": "string",
            "description": "This is returned in the case of a password reset."
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /dsltest

Check whether DSL code fragments are correct.

Description

This endpoint will execute a provided DSL fragment and return the result. It will be run with test data, but you can use it to call your models or whatever. Queries that take too long will be terminated.

DSL Phases

There are three places where the DSL is used:

  • Sending requests to models
  • Parsing the responses that models return
  • Grading the parsed responses

These three steps happen sequentially for each task. This endpoint only checks one phase, which you must specify. That being said, there's nothing stopping you from chaining all three, e.g.:

import requests

API_KEY = "<your api key goes here>"

def run_code(code, stage, **overrides):
    headers = {'Api-Token': API_KEY}
    res = requests.post('https://equistamp.net/dsltest', headers=headers,
json={"code": code, "stage": stage, **overrides})
    if res.status_code != 200:
        raise ValueError(f'bad request: {res.text}')
    return res.json()

response = run_code('(POST "https://your.model/endpoint" {:json {"task"
task}})', 'request')
parsed_response = run_code('(get-in response ["path" "to" "response"])',
'response', response=response)
grader_result = run_code('parsed-response', 'grader', response=response,
parsed_response=parsed_response)

print(grader_result)

Context

When starting a request, a context is created with useful constants:

Base constants

  • task - the text of the task to be completed
  • prompt - the task text wrapped in a prompt for LLMs
  • endpoint_type - the type of endpoint - possible values are: aws, google_cloud, anthropic, text2text-generation, text-generation, azure, zero- shot-classification, open_ai, mistral, fill-mask, custom, conversational, together.ai
  • cache - An atom containing a cache that can be used to store data between requests. Acts as a map, so items can be accessed via (get @cache <key>) and set via (swap! cache assoc <key> <val>).

Task specific context

Mulitple choice tasks

In the case of multiple choice tasks, the following are also available:

  • correct - the letters of all correct answers
  • num_choices - the number of available choices
  • letter-choices - the letters corresponding to the available choices

Stage context

Each subsequent stage (response, grader) will have values added in the previous stages:

Response
  • response - the result of the Request DSL call
Grader
  • parsed-response - the result of the Response call

Request body

{
    "code": "(get-in response [:json \"value\"])",
    "stage": "response",
    "response": {
        "json": {
            "value": "bla bla"
        }
    },
    "parsed_response": "bla bla"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "code": {
            "description": "The DSL code to be evaluated",
            "type": "string",
            "example": "(get-in response [:json \"value\"])"
        },
        "stage": {
            "description": "The kind of DSL code to be tested",
            "example": "response",
            "type": "string",
            "enum": [
                "request",
                "response",
                "grader"
            ]
        },
        "response": {
            "description": "The response used when testing 'response' DSL code. If not provided, a dummy value will be used",
            "example": {
                "json": {
                    "value": "bla bla"
                }
            }
        },
        "parsed_response": {
            "description": "The parsed_response used when testing 'grader' DSL code. If not provided, a dummy value will be used",
            "example": "bla bla"
        }
    }
}

Response 200 OK 

{
    "result": null
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "result": {
            "description": "This will be whatever the code returned"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /evaluation

Create a new evaluation.

Description

Adding tasks to new evaluations

There are three ways to add tasks to evaluations:

  1. directly during creation by providing a CSV with tasks via the csv_url and columns_mapping parameters
  2. by sending a tasks CSV to the /evaluationbuilderhandler endpoint
  3. by uploading tasks directly via the /task endpoint

The first option is recommended, as it will automatically call the /evaluationbuilderhandler endpoint for you, once the evaluation is created.

Request body

{
    "name": "string",
    "public": true,
    "num_tasks": 10.12,
    "description": "string",
    "task_types": [
        "string"
    ],
    "modalities": [
        "string"
    ],
    "min_questions_to_complete": 10.12,
    "csv_url": "https://example.com",
    "default_task_type": "MCQ",
    "columns_mapping": {
        "Question col": {
            "columnType": "question"
        },
        "Paraphrase of question": {
            "columnType": "paraphrase",
            "paraphraseOf": "Question col"
        }
    }
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "public": {
            "type": "boolean"
        },
        "num_tasks": {
            "type": "number",
            "format": "int64"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "task_types": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "min_questions_to_complete": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "csv_url": {
            "description": "The URL of a CSV file containing the tasks of the new evaluation",
            "example": "https://example.com",
            "type": "string"
        },
        "default_task_type": {
            "description": "The default type of tasks - can be overrode on a per row basis. Will use \"MCQ\" if not set",
            "example": "MCQ",
            "nullable": true,
            "type": "string",
            "enum": [
                "MCQ",
                "FRQ"
            ]
        },
        "columns_mapping": {
            "description": "A mapping that specifies which CSV columns contain which types of data. See the [Evaluation Builder](#post-evaluationbuilderhandler) endpoint for details",
            "type": "object",
            "example": {
                "Question col": {
                    "columnType": "question"
                },
                "Paraphrase of question": {
                    "columnType": "paraphrase",
                    "paraphraseOf": "Question col"
                }
            },
            "additionalProperties": {
                "$ref": "#/components/schemas/ColumnMapping"
            }
        }
    }
}

Response 201 Created 

{
    "id": "9f4e22fc-9b59-4100-8c2e-6b1089819f34",
    "name": "string",
    "public": true,
    "num_tasks": 10.12,
    "description": "string",
    "last_updated": "2022-04-13T15:42:05.901Z",
    "task_types": [
        "string"
    ],
    "modalities": [
        "string"
    ],
    "min_questions_to_complete": 10.12,
    "owner": {
        "id": "f6fd0633-6edc-4213-afec-f56ec9ec4b3b",
        "user_name": "string",
        "full_name": "string",
        "user_image": "string",
        "bio": "string",
        "display_options": null,
        "join_date": "2022-04-13",
        "email_address": "string",
        "subscription_level": "string",
        "alarms": [
            "157d8e4d-7c24-44a3-9c0b-2d9aafca7a47"
        ]
    }
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "name": {
            "type": "string"
        },
        "public": {
            "type": "boolean"
        },
        "num_tasks": {
            "type": "number",
            "format": "int64"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "last_updated": {
            "type": "string",
            "format": "date-time"
        },
        "task_types": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "min_questions_to_complete": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "owner": {
            "$ref": "#/components/schemas/ShallowUser"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /evaluation

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/Evaluation"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/Evaluation"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /evaluation

Request body

{
    "name": "string",
    "public": true,
    "num_tasks": 10.12,
    "description": "string",
    "task_types": [
        "string"
    ],
    "modalities": [
        "string"
    ],
    "min_questions_to_complete": 10.12
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "public": {
            "type": "boolean"
        },
        "num_tasks": {
            "type": "number",
            "format": "int64"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "task_types": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "min_questions_to_complete": {
            "type": "number",
            "format": "int64",
            "nullable": true
        }
    }
}

Response 200 OK 

"Evaluation updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "Evaluation updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /evaluationbuilderhandler

Import tasks from a CSV file.

Description

This endpoint will fetch a CSV file and create a task from each row (without the first one, which is used as a header).

Number of questions to complete

Each evaluation run will use a subsample of all available tasks. You can set this number by providing a value for min_questions_to_complete. If you don't set this manually, it will be set on the basis of the number of tasks in your file, in such a way as to have a 95% confidence level. In practice this number tends to be larger than needed - the score of most evaluation runs don't change that much after around 200 tasks.

Task type

Unless specified otherwise, it's assumed that all tasks are Multiple Choice Questions. The can be changed by

  1. setting default_task_type, which will change the default to whatever you provide
  2. providing a type column, which can be used to set the task types for specific rows - any rows where the type column is not empty will that value as the type, otherwise will use the default type
Columns mapping

For the CSV import to work correctly, you must provide a way to map columns to task fields. This is done by providing a mapping of <column name> to a column definition object. The available fields in the definition object are:

  • columnType - this specified what this column should be used as. Must always be provided
  • paraphraseOf - used by paraphrase columns to point to what they're paraphrasing. All texts can have paraphrases. When a field has paraphrases defined, these will always be used when sending texts to models, or displaying them on the frontend. Only you and system administrators will have access to the non paraphrase texts.

Request body

{
    "min_questions_to_complete": 10.12,
    "evaluation_id": "83ef18ef-debf-4960-b19f-e0ccde87ef2a",
    "csv_url": "https://example.com",
    "default_task_type": "MCQ",
    "columns_mapping": {
        "Question col": {
            "columnType": "question"
        },
        "Paraphrase of question": {
            "columnType": "paraphrase",
            "paraphraseOf": "Question col"
        }
    }
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "min_questions_to_complete": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "evaluation_id": {
            "description": "The id of the evaluation to add tasks to",
            "type": "string",
            "format": "uuid"
        },
        "csv_url": {
            "description": "The URL of a CSV file containing the tasks of the new evaluation",
            "example": "https://example.com",
            "type": "string"
        },
        "default_task_type": {
            "description": "The default type of tasks - can be overrode on a per row basis. Will use \"MCQ\" if not set",
            "example": "MCQ",
            "nullable": true,
            "type": "string",
            "enum": [
                "MCQ",
                "FRQ"
            ]
        },
        "columns_mapping": {
            "description": "A mapping that specifies which CSV columns contain which types of data. See the [Evaluation Builder](#post-evaluationbuilderhandler) endpoint for details",
            "type": "object",
            "example": {
                "Question col": {
                    "columnType": "question"
                },
                "Paraphrase of question": {
                    "columnType": "paraphrase",
                    "paraphraseOf": "Question col"
                }
            },
            "additionalProperties": {
                "$ref": "#/components/schemas/ColumnMapping"
            }
        }
    }
}

Response 201 Created 

{
    "id": "3ac06e6f-9bf8-481b-a643-63d6fb2f7923",
    "name": "string",
    "public": true,
    "num_tasks": 10.12,
    "description": "string",
    "last_updated": "2022-04-13T15:42:05.901Z",
    "task_types": [
        "string"
    ],
    "modalities": [
        "string"
    ],
    "min_questions_to_complete": 10.12,
    "owner": {
        "id": "83c67ca4-0e16-4132-b520-a6669cafc9c7",
        "user_name": "string",
        "full_name": "string",
        "user_image": "string",
        "bio": "string",
        "display_options": null,
        "join_date": "2022-04-13",
        "email_address": "string",
        "subscription_level": "string",
        "alarms": [
            "0da5cf47-7e47-4b99-acdf-ba8cc728cde4"
        ]
    }
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "name": {
            "type": "string"
        },
        "public": {
            "type": "boolean"
        },
        "num_tasks": {
            "type": "number",
            "format": "int64"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "last_updated": {
            "type": "string",
            "format": "date-time"
        },
        "task_types": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "min_questions_to_complete": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "owner": {
            "$ref": "#/components/schemas/ShallowUser"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /evaluationbuilderhandler

Check whether a CSV file contains valid tasks

Description

This endpoint will fetch a CSV file from the provided URL and validate each row to make sure that it can be processed. Rows with errors or warnings will be returned with appropriate messages, to help debug problems. When the CSV is processed (after sending an appropriate POST request to this endpoint), rows that have errors will be skipped.

Column mapping

To check whether all the rows are correct, you must provide a way to work out which columns correspond to which fields in the resulting tasks. In the case of GET requests, they should be provided as follows. Check out our sample tasks file for examples:

Basic mappings

  • question - this is the only required parameter. This should specify the name of the column containing the main text to be sent to models
  • type - this specifies where to check for per row task type overrides. By default it's assumed that tasks are multiple choice questions, unless default_task_type is set in the POST request. But if you want most tasks to be one type, but have a couple that are of a different type (e.g. true-false questions), then you can do so by using this column.
  • redacted - this specified where to check whether a task should be hidden by default. By default it's assumed that all tasks should be used when testing models, but sometimes a given task may be incorrect, or maybe not the best quality. One way around this would be to delete any problematical rows before uploading, but that can be a lot of work. To make things easier, tasks can be uploaded as redacted, which means that they won't be sent to models. Any rows with a redacted column defined, which have non empty values, will be saved as redacted

Paraphrases

All texts can have paraphrases. When a field has paraphrases defined, these will always be used when sending texts to models, or displaying them on the frontend. Only you and system administrators will have access to the non paraphrase texts. Paraphrases are declared as paraphrase.<paraphrase column>=<paraphrased column>. So e.g. paraphrase.question%20paraphrase=Question will declare that the "question paraphrase" column is a paraphrase of the "Question" column.

Multiple response question mappings

In the case of multiple response questions, you must provide at least one correct answer, and at least one incorrect answers. You can add more if you want, but we will only use the first 10 correct answers, and the first 20 incorrect answers. These column definitions should be provided via:

  • mcq_correct - a comma separated list of URL encoded column names, e.g. 'Correct%201,Correct%20%3D%20this'
  • mcq_incorrect - a comma separated list of URL encoded column names, e.g. 'This%20is%20wrong,Bad%21%21'

Example column mappings

Assuming you have a CSV file with the following columns:

  • Task type - contains the type of tasks
  • Timestamp - date of last edit - not needed here, so should be ignored
  • `` - an empty column
  • Task question to answer - the text to which models should respond
  • Question paraphrase - an alternative way of phrasing the question
  • Correct answer - the expected answer
  • Alternative correct answer - another answer that will also be accepted as correct
  • Bad response example - an incorrect answer to be provided as an option in the multiple choice question
  • Wrong answer - another incorrect answer to be provided as an option in the multiple choice question

The you would have to send a GET request with type=Task%20type&question=Tas k%20question%20to%20answer&paraphrase.Question%20paraphrase=Task question to answer&mcq_correct=Correct%20answer,Alternative%20correct%20answer&mcq_incor rect=Bad%20response%20example,Wrong%20answer

Input parameters

Parameter In Type Default Nullable Description
csv_url path None No The URL of a CSV file containing the tasks of the new evaluation
only_header path None No When set, will just return the headers of the CSV file
question path None No The column in the CSV file containing the questions
redacted path None No The column in the CSV file marking tasks as redacted
type path None No The column in the CSV file containing the per row task type

Response 200 OK 

{
    "errors": [
        {
            "task_num": 3,
            "errors": [
                "This row couldn't be parsed"
            ],
            "warnings": [
                "This row is suspicious"
            ]
        }
    ],
    "num_tasks": 123,
    "min_questions_to_complete": 42
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "errors": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "task_num": {
                        "description": "The index of the row that has these errors",
                        "type": "number",
                        "format": "int64",
                        "example": 3
                    },
                    "errors": {
                        "type": "array",
                        "items": {
                            "type": "string",
                            "example": "This row couldn't be parsed"
                        }
                    },
                    "warnings": {
                        "type": "array",
                        "items": {
                            "type": "string",
                            "example": "This row is suspicious"
                        }
                    }
                }
            }
        },
        "num_tasks": {
            "description": "The number of rows with tasks found, including rows with errors",
            "type": "number",
            "format": "int64",
            "example": 123
        },
        "min_questions_to_complete": {
            "description": "The minimum number of tasks per evaluation session. If this wasn't provided in the query parameters, it will be calculated based on the number of tasks found",
            "type": "number",
            "format": "int64",
            "example": 42
        }
    }
}

Response 400 Bad Request

Refer to the common response description: ValidationError.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /evaluationmodeljobshandler

Request body

{
    "job_name": "string",
    "minutes_between_evaluations": 10.12,
    "job_body": null,
    "job_description": "string",
    "start_date": "2022-04-13T15:42:05.901Z",
    "model_id": "0e88e5b6-e93f-49b0-9a5e-78e8a944b6f7",
    "evaluation_id": "66418034-f032-4985-a2ad-6730e4affab0"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "job_name": {
            "type": "string"
        },
        "minutes_between_evaluations": {
            "type": "number",
            "format": "int64"
        },
        "job_body": {},
        "job_description": {
            "type": "string"
        },
        "start_date": {
            "type": "string",
            "format": "date-time",
            "nullable": true
        },
        "model_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 201 Created 

{
    "job_name": "string",
    "minutes_between_evaluations": 10.12,
    "job_body": null,
    "job_description": "string",
    "job_schedule_arn": "string",
    "start_date": "2022-04-13T15:42:05.901Z",
    "owner_id": "cb88cad6-c50e-4d33-a8b7-5a00d5d684e9",
    "model_id": "f31d039e-37ed-48c9-8887-8444dfd72380",
    "evaluation_id": "7b65452d-64d6-40a6-b4d0-91e18c7278d8",
    "id": "87ad942d-03aa-43e6-bfe8-0726fe4ca62d",
    "creation_date": "2022-04-13T15:42:05.901Z"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "job_name": {
            "type": "string"
        },
        "minutes_between_evaluations": {
            "type": "number",
            "format": "int64"
        },
        "job_body": {},
        "job_description": {
            "type": "string"
        },
        "job_schedule_arn": {
            "type": "string"
        },
        "start_date": {
            "type": "string",
            "format": "date-time",
            "nullable": true
        },
        "owner_id": {
            "type": "string",
            "format": "uuid"
        },
        "model_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        },
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "creation_date": {
            "type": "string",
            "format": "date-time"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /evaluationsession

Request body

{
    "datetime_completed": "2022-04-13T15:42:05.901Z",
    "origin": "string",
    "completed": true,
    "is_human_being_evaluated": true,
    "num_questions_answered": 10.12,
    "num_answered_correctly": 10.12,
    "median_seconds_per_task": 10.12,
    "min_seconds_per_task": 10.12,
    "max_seconds_per_task": 10.12,
    "session_task_orchestrator_schedule_arn": "string",
    "evaluatee_id": "d3700e97-54d2-42b0-b326-bb5c27e3e233",
    "evaluation_id": "d7a79bc2-c39c-4965-9712-4e56a99b1751",
    "restart": true
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "datetime_completed": {
            "type": "string",
            "format": "date-time",
            "nullable": true
        },
        "origin": {
            "type": "string"
        },
        "completed": {
            "type": "boolean"
        },
        "is_human_being_evaluated": {
            "type": "boolean"
        },
        "num_questions_answered": {
            "type": "number",
            "format": "int64"
        },
        "num_answered_correctly": {
            "type": "number",
            "format": "int64"
        },
        "median_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "min_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "max_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "session_task_orchestrator_schedule_arn": {
            "type": "string",
            "nullable": true
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        },
        "restart": {
            "description": "Will force a new evaluation session if true - by default, calling this endpoint for a evaluation - model session that is already running, will add more tasks to the running session, rather than creating a new one",
            "type": "boolean"
        }
    }
}

Response 201 Created 

{
    "id": "ae531545-27f8-480f-932a-1ac85cfb2003",
    "datetime_started": "2022-04-13T15:42:05.901Z",
    "datetime_completed": "2022-04-13T15:42:05.901Z",
    "origin": "string",
    "completed": true,
    "failed": true,
    "is_human_being_evaluated": true,
    "num_questions_answered": 10.12,
    "num_answered_correctly": 10.12,
    "num_endpoint_failures": 10.12,
    "num_endpoint_calls": 10.12,
    "num_characters_sent_to_endpoint": 10.12,
    "num_characters_received_from_endpoint": 10.12,
    "estimated_session_cost_usd": 10.12,
    "median_seconds_per_task": 10.12,
    "mean_seconds_per_task": 10.12,
    "std_seconds_per_task": 10.12,
    "distribution_of_seconds_per_task": null,
    "min_seconds_per_task": 10.12,
    "max_seconds_per_task": 10.12,
    "median_characters_per_task": 10.12,
    "mean_characters_per_task": 10.12,
    "std_characters_per_task": 10.12,
    "distribution_of_characters_per_task": null,
    "min_characters_per_task": 10.12,
    "max_characters_per_task": 10.12,
    "evaluatee_id": "2c470dd9-c73d-4043-b48c-707a8e87b562",
    "evaluation_id": "1f726140-a137-4a40-b635-b95540d2e568"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "datetime_started": {
            "type": "string",
            "format": "date-time"
        },
        "datetime_completed": {
            "type": "string",
            "format": "date-time",
            "nullable": true
        },
        "origin": {
            "type": "string"
        },
        "completed": {
            "type": "boolean"
        },
        "failed": {
            "type": "boolean"
        },
        "is_human_being_evaluated": {
            "type": "boolean"
        },
        "num_questions_answered": {
            "type": "number",
            "format": "int64"
        },
        "num_answered_correctly": {
            "type": "number",
            "format": "int64"
        },
        "num_endpoint_failures": {
            "type": "number",
            "format": "int64"
        },
        "num_endpoint_calls": {
            "type": "number",
            "format": "int64"
        },
        "num_characters_sent_to_endpoint": {
            "type": "number",
            "format": "int64"
        },
        "num_characters_received_from_endpoint": {
            "type": "number",
            "format": "int64"
        },
        "estimated_session_cost_usd": {
            "type": "number",
            "format": "double"
        },
        "median_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "mean_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "std_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "distribution_of_seconds_per_task": {
            "nullable": true
        },
        "min_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "max_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "median_characters_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "mean_characters_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "std_characters_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "distribution_of_characters_per_task": {
            "nullable": true
        },
        "min_characters_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "max_characters_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /evaluationsession

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/EvaluationSession"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/EvaluationSession"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /evaluationsession

Request body

{
    "datetime_completed": "2022-04-13T15:42:05.901Z",
    "origin": "string",
    "completed": true,
    "is_human_being_evaluated": true,
    "num_questions_answered": 10.12,
    "num_answered_correctly": 10.12,
    "median_seconds_per_task": 10.12,
    "min_seconds_per_task": 10.12,
    "max_seconds_per_task": 10.12,
    "session_task_orchestrator_schedule_arn": "string",
    "evaluatee_id": "71338b53-46cf-45a9-bde2-e8cec5dedd5a",
    "evaluation_id": "8da76287-4445-4339-b6ff-e6ff8d3e671f"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "datetime_completed": {
            "type": "string",
            "format": "date-time",
            "nullable": true
        },
        "origin": {
            "type": "string"
        },
        "completed": {
            "type": "boolean"
        },
        "is_human_being_evaluated": {
            "type": "boolean"
        },
        "num_questions_answered": {
            "type": "number",
            "format": "int64"
        },
        "num_answered_correctly": {
            "type": "number",
            "format": "int64"
        },
        "median_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "min_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "max_seconds_per_task": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "session_task_orchestrator_schedule_arn": {
            "type": "string",
            "nullable": true
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 200 OK 

"EvaluationSession updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "EvaluationSession updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /model

Request body

{
    "name": "string",
    "description": "string",
    "publisher": "string",
    "architecture": "string",
    "picture": "string",
    "num_parameters": 10.12,
    "modalities": [
        "string"
    ],
    "public": true,
    "public_usable": true,
    "endpoint_type": "string",
    "setup_code": "string",
    "teardown_code": "string",
    "request_code": "string",
    "response_code": "string",
    "task_holding_queue_url": "string",
    "task_execution_queue_url": "string",
    "task_execution_dlq_url": "string",
    "lambda_arn": "string",
    "cost_per_input_character_usd": 10.12,
    "cost_per_output_character_usd": 10.12,
    "cost_per_instance_hour_usd": 10.12,
    "max_characters_per_minute": 10.12,
    "max_request_per_minute": 10.12
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "publisher": {
            "type": "string",
            "nullable": true
        },
        "architecture": {
            "type": "string",
            "nullable": true
        },
        "picture": {
            "type": "string",
            "nullable": true
        },
        "num_parameters": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "public": {
            "type": "boolean"
        },
        "public_usable": {
            "type": "boolean"
        },
        "endpoint_type": {
            "type": "string"
        },
        "setup_code": {
            "type": "string",
            "nullable": true
        },
        "teardown_code": {
            "type": "string",
            "nullable": true
        },
        "request_code": {
            "type": "string",
            "nullable": true
        },
        "response_code": {
            "type": "string",
            "nullable": true
        },
        "task_holding_queue_url": {
            "type": "string",
            "nullable": true
        },
        "task_execution_queue_url": {
            "type": "string",
            "nullable": true
        },
        "task_execution_dlq_url": {
            "type": "string",
            "nullable": true
        },
        "lambda_arn": {
            "type": "string",
            "nullable": true
        },
        "cost_per_input_character_usd": {
            "type": "number",
            "format": "double"
        },
        "cost_per_output_character_usd": {
            "type": "number",
            "format": "double"
        },
        "cost_per_instance_hour_usd": {
            "type": "number",
            "format": "double"
        },
        "max_characters_per_minute": {
            "type": "number",
            "format": "int64"
        },
        "max_request_per_minute": {
            "type": "number",
            "format": "int64"
        }
    }
}

Response 201 Created 

{
    "id": "5e571faa-fadf-4ee8-bcee-a684ed02dda4",
    "name": "string",
    "description": "string",
    "owner_id": "f7b75b91-b871-4fbc-9e93-3a967c47db6f",
    "publisher": "string",
    "architecture": "string",
    "picture": "string",
    "num_parameters": 10.12,
    "modalities": [
        "string"
    ],
    "public": true,
    "public_usable": true,
    "endpoint_type": "string",
    "max_characters_per_minute": 10.12,
    "max_request_per_minute": 10.12,
    "score": 10.12,
    "top_example_id": "6201b6d0-1693-4368-afc8-426fa3bfd09f",
    "worst_example_id": "5d1b03e2-74a0-49c8-a779-ee48151af92c",
    "owner": {
        "id": "8eca2f1c-c3c9-4985-b71f-ebc4cde9cb54",
        "user_name": "string",
        "full_name": "string",
        "user_image": "string",
        "bio": "string",
        "display_options": null,
        "join_date": "2022-04-13",
        "email_address": "string",
        "subscription_level": "string",
        "alarms": [
            "46141def-304b-4136-bbb8-c6136cc576c6"
        ]
    },
    "top_example": {
        "id": "d3425760-890d-444b-8d3a-32cd8a84dbd8",
        "task_type": "string",
        "is_task_live": true,
        "modalities": [
            "string"
        ],
        "redacted": true,
        "num_possible_answers": 10.12,
        "evaluation_task_number": 10.12,
        "median_human_completion_seconds": 10.12,
        "median_ai_completion_seconds": 10.12,
        "num_times_human_evaluated": 10.12,
        "num_times_ai_evaluated": 10.12,
        "num_times_humans_answered_correctly": 10.12,
        "num_times_ai_answered_correctly": 10.12,
        "evaluation_id": "fff9298a-7f43-443b-83ab-00663efe35d0",
        "owner_id": "39810f32-f362-4a3f-8d0d-bc43b2ac2aad",
        "tags": [
            "8e0533c5-e24a-44ac-9782-f71f625440f0"
        ]
    },
    "worst_example": null,
    "best_eval": {
        "id": "b370531f-755c-4562-a65d-743a3d103a92",
        "datetime_started": "2022-04-13T15:42:05.901Z",
        "datetime_completed": "2022-04-13T15:42:05.901Z",
        "origin": "string",
        "completed": true,
        "failed": true,
        "is_human_being_evaluated": true,
        "num_questions_answered": 10.12,
        "num_answered_correctly": 10.12,
        "num_endpoint_failures": 10.12,
        "num_endpoint_calls": 10.12,
        "num_characters_sent_to_endpoint": 10.12,
        "num_characters_received_from_endpoint": 10.12,
        "estimated_session_cost_usd": 10.12,
        "median_seconds_per_task": 10.12,
        "mean_seconds_per_task": 10.12,
        "std_seconds_per_task": 10.12,
        "distribution_of_seconds_per_task": null,
        "min_seconds_per_task": 10.12,
        "max_seconds_per_task": 10.12,
        "median_characters_per_task": 10.12,
        "mean_characters_per_task": 10.12,
        "std_characters_per_task": 10.12,
        "distribution_of_characters_per_task": null,
        "min_characters_per_task": 10.12,
        "max_characters_per_task": 10.12,
        "evaluatee_id": "681f77ab-c653-435a-b41f-af5834181996",
        "evaluation_id": "f8104431-b7ea-4521-9ee3-3f1a1f9f8327"
    },
    "worst_eval": null
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "name": {
            "type": "string"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "owner_id": {
            "type": "string",
            "format": "uuid"
        },
        "publisher": {
            "type": "string",
            "nullable": true
        },
        "architecture": {
            "type": "string",
            "nullable": true
        },
        "picture": {
            "type": "string",
            "nullable": true
        },
        "num_parameters": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "public": {
            "type": "boolean"
        },
        "public_usable": {
            "type": "boolean"
        },
        "endpoint_type": {
            "type": "string"
        },
        "max_characters_per_minute": {
            "type": "number",
            "format": "int64"
        },
        "max_request_per_minute": {
            "type": "number",
            "format": "int64"
        },
        "score": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "top_example_id": {
            "type": "string",
            "format": "uuid",
            "nullable": true
        },
        "worst_example_id": {
            "type": "string",
            "format": "uuid",
            "nullable": true
        },
        "owner": {
            "$ref": "#/components/schemas/ShallowUser"
        },
        "top_example": {
            "$ref": "#/components/schemas/ShallowTask"
        },
        "worst_example": {
            "$ref": "#/components/schemas/ShallowTask"
        },
        "best_eval": {
            "$ref": "#/components/schemas/ShallowEvaluationSession"
        },
        "worst_eval": {
            "$ref": "#/components/schemas/ShallowEvaluationSession"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /model

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/Model"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/Model"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /model

Request body

{
    "name": "string",
    "description": "string",
    "publisher": "string",
    "architecture": "string",
    "picture": "string",
    "num_parameters": 10.12,
    "modalities": [
        "string"
    ],
    "public": true,
    "public_usable": true,
    "endpoint_type": "string",
    "setup_code": "string",
    "teardown_code": "string",
    "request_code": "string",
    "response_code": "string",
    "task_holding_queue_url": "string",
    "task_execution_queue_url": "string",
    "task_execution_dlq_url": "string",
    "lambda_arn": "string",
    "cost_per_input_character_usd": 10.12,
    "cost_per_output_character_usd": 10.12,
    "cost_per_instance_hour_usd": 10.12,
    "max_characters_per_minute": 10.12,
    "max_request_per_minute": 10.12
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "name": {
            "type": "string"
        },
        "description": {
            "type": "string",
            "nullable": true
        },
        "publisher": {
            "type": "string",
            "nullable": true
        },
        "architecture": {
            "type": "string",
            "nullable": true
        },
        "picture": {
            "type": "string",
            "nullable": true
        },
        "num_parameters": {
            "type": "number",
            "format": "int64",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "public": {
            "type": "boolean"
        },
        "public_usable": {
            "type": "boolean"
        },
        "endpoint_type": {
            "type": "string"
        },
        "setup_code": {
            "type": "string",
            "nullable": true
        },
        "teardown_code": {
            "type": "string",
            "nullable": true
        },
        "request_code": {
            "type": "string",
            "nullable": true
        },
        "response_code": {
            "type": "string",
            "nullable": true
        },
        "task_holding_queue_url": {
            "type": "string",
            "nullable": true
        },
        "task_execution_queue_url": {
            "type": "string",
            "nullable": true
        },
        "task_execution_dlq_url": {
            "type": "string",
            "nullable": true
        },
        "lambda_arn": {
            "type": "string",
            "nullable": true
        },
        "cost_per_input_character_usd": {
            "type": "number",
            "format": "double"
        },
        "cost_per_output_character_usd": {
            "type": "number",
            "format": "double"
        },
        "cost_per_instance_hour_usd": {
            "type": "number",
            "format": "double"
        },
        "max_characters_per_minute": {
            "type": "number",
            "format": "int64"
        },
        "max_request_per_minute": {
            "type": "number",
            "format": "int64"
        }
    }
}

Response 200 OK 

"Model updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "Model updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /modelsconnecter

Request body

{
    "evaluation_id": "81271575-39f9-45cf-a87a-8bcfa8adc115",
    "evaluatee_id": "2159e284-a104-47e1-b7e7-0c1783cf3956",
    "cadence": "string",
    "price": 10.12
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "cadence": {
            "type": "string",
            "nullable": true
        },
        "price": {
            "type": "number",
            "format": "int64"
        }
    }
}

Response 201 Created 

{
    "id": "06e8241b-9c83-4894-92aa-1599b756d798",
    "evaluation_id": "d0973798-3391-479e-8ce0-5c8b8b847253",
    "evaluatee_id": "84309557-7f48-43f5-ab04-1ad4edc69d44",
    "status": "string",
    "cadence": "string",
    "price": 10.12,
    "model": {
        "id": "d15c93df-e2ad-4010-82f4-b8697972f9b4",
        "name": "string",
        "description": "string",
        "owner_id": "0791d10b-4439-4fdc-a52b-3fe10af15875",
        "publisher": "string",
        "architecture": "string",
        "picture": "string",
        "num_parameters": 10.12,
        "modalities": [
            "string"
        ],
        "public": true,
        "public_usable": true,
        "endpoint_type": "string",
        "max_characters_per_minute": 10.12,
        "max_request_per_minute": 10.12,
        "score": 10.12,
        "top_example_id": "b788fec3-3696-4057-90e7-9c192dbfdeb5",
        "worst_example_id": "c4e44200-e125-4985-af45-82b62f8370d7",
        "owner": "d47371ca-4704-4139-9971-a1f40025984f",
        "top_example": "333ffb51-df6e-48cf-97ec-08d37b1e5bcd",
        "worst_example": "e53c392d-d7f2-43f8-aa31-733b094d8944",
        "best_eval": "1a81c7c5-774b-47f8-aeb7-e0c34c145dc4",
        "worst_eval": "9c46e788-dcfc-492c-a66e-544c60a449d7"
    }
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "status": {
            "type": "string"
        },
        "cadence": {
            "type": "string",
            "nullable": true
        },
        "price": {
            "type": "number",
            "format": "int64"
        },
        "model": {
            "$ref": "#/components/schemas/ShallowModel"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /modelsconnecter

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/EvaluationEvaluatee"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/EvaluationEvaluatee"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /queryexternalmodelhandler

Request body

{
    "response_time_in_seconds": 10.12,
    "task_id": "cea18c01-92dd-4619-8576-99d77ac81bb5",
    "evaluation_session_id": "38cd40f5-90e8-4d38-b4e6-becdfc08950f",
    "model_id": "213161cb-f6ea-4aa6-a662-48bf687f64ea"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "response_time_in_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "task_id": {
            "description": "The id of the task to be run on the model",
            "type": "string",
            "format": "uuid"
        },
        "evaluation_session_id": {
            "description": "The id of the evaluation session that is being checked",
            "type": "string",
            "format": "uuid"
        },
        "model_id": {
            "description": "The id of the model that is being evaluation",
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 201 Created 

{
    "id": "8d9374e6-141b-46a6-929b-cdaa612b1913",
    "correct": true,
    "raw_task_text": "string",
    "raw_response_text": "string",
    "parsed_response_text": "string",
    "response_time_in_seconds": 10.12,
    "correctness": 10.12,
    "task_id": "074ab40a-3062-402b-8c8c-5d787c988fa8",
    "evaluatee_id": "165f74cb-017d-4950-bbb5-524e915acf51",
    "chosen_answer_id": "8e5f9049-e95f-4e0f-aeff-3560bdcfe41e",
    "evaluation_session_id": "306d494b-91c9-45fe-b4cf-fb7a2219072f",
    "creation_date": "2022-04-13T15:42:05.901Z"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "correct": {
            "type": "boolean",
            "nullable": true
        },
        "raw_task_text": {
            "type": "string",
            "nullable": true
        },
        "raw_response_text": {
            "type": "string",
            "nullable": true
        },
        "parsed_response_text": {
            "type": "string",
            "nullable": true
        },
        "response_time_in_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "correctness": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "task_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "chosen_answer_id": {
            "type": "string",
            "format": "uuid",
            "nullable": true
        },
        "evaluation_session_id": {
            "type": "string",
            "format": "uuid"
        },
        "creation_date": {
            "type": "string",
            "format": "date-time"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /response

Request body

{
    "response_time_in_seconds": 10.12,
    "task_id": "550f1d12-3cbd-42f3-8f58-b5e1c60e6761",
    "evaluation_session_id": "3de45256-52dd-46d0-9875-fba3a096ea17",
    "task_type": "MCQ",
    "question": "What time is it?",
    "answer_text": "Half past nine",
    "answer_id": "101d2ee2-16a1-456f-b0e6-813890b3ee74"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "response_time_in_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "task_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_session_id": {
            "type": "string",
            "format": "uuid"
        },
        "task_type": {
            "description": "The type of tasks for which this is a response",
            "example": "MCQ",
            "type": "string",
            "enum": [
                "MCQ",
                "FRQ"
            ]
        },
        "question": {
            "type": "string",
            "description": "The text of the question for which this is a response",
            "example": "What time is it?"
        },
        "answer_text": {
            "type": "string",
            "description": "The text returned from the model",
            "example": "Half past nine"
        },
        "answer_id": {
            "type": "string",
            "format": "uuid",
            "nullable": true,
            "description": "The id of the selected answer, in the case of multiple choice questions"
        }
    }
}

Response 201 Created 

{
    "id": "fc9a8a9a-912a-4f03-a8da-a9ba814418f9",
    "correct": true,
    "raw_task_text": "string",
    "raw_response_text": "string",
    "parsed_response_text": "string",
    "response_time_in_seconds": 10.12,
    "correctness": 10.12,
    "task_id": "c75680e0-611a-470e-aa4a-4ac4e3920e58",
    "evaluatee_id": "891f7f28-6678-472f-9ca9-bddd9319c743",
    "chosen_answer_id": "b6287bc3-504c-4bb0-bbd2-278c7a89249c",
    "evaluation_session_id": "8b093979-8768-4823-8b44-e3c2d2bb6c9f",
    "creation_date": "2022-04-13T15:42:05.901Z"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "correct": {
            "type": "boolean",
            "nullable": true
        },
        "raw_task_text": {
            "type": "string",
            "nullable": true
        },
        "raw_response_text": {
            "type": "string",
            "nullable": true
        },
        "parsed_response_text": {
            "type": "string",
            "nullable": true
        },
        "response_time_in_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "correctness": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "task_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluatee_id": {
            "type": "string",
            "format": "uuid"
        },
        "chosen_answer_id": {
            "type": "string",
            "format": "uuid",
            "nullable": true
        },
        "evaluation_session_id": {
            "type": "string",
            "format": "uuid"
        },
        "creation_date": {
            "type": "string",
            "format": "date-time"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /response

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/Response"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/Response"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /response

Request body

{
    "response_time_in_seconds": 10.12,
    "task_id": "e3836a8b-525e-4e95-ae71-e05ca5524f73",
    "evaluation_session_id": "e560cb06-9d57-4a1f-957e-a9a1491f5905"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "response_time_in_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "task_id": {
            "type": "string",
            "format": "uuid"
        },
        "evaluation_session_id": {
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 200 OK 

"Response updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "Response updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /tag

Request body

Schema of the request body 
{
    "type": "object",
    "properties": {}
}

Response 201 Created 

{
    "id": "4d72a654-1355-4318-9d32-278945fbb508",
    "name": "string"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "name": {
            "type": "string"
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /tag

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/Tag"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/Tag"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /tag

Request body

Schema of the request body 
{
    "type": "object",
    "properties": {}
}

Response 200 OK 

"Tag updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "Tag updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /task

Request body

{
    "task_type": "string",
    "is_task_live": true,
    "modalities": [
        "string"
    ],
    "redacted": true,
    "tags": [
        "3220a788-32ea-4657-ac6a-b8023ec81375"
    ],
    "type": "MCQ",
    "question": "What time is it?",
    "answers": [
        {
            "text": "half past one",
            "paraphrases": [
                "1:30 PM",
                "13:30"
            ],
            "correct": false
        },
        {
            "text": "Time is an illusion",
            "correct": false
        },
        {
            "text": "Now",
            "correct": true
        }
    ],
    "evaluation_id": "cd4f27b9-77ec-4191-afa3-e9aa90f1df98"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "task_type": {
            "type": "string"
        },
        "is_task_live": {
            "type": "boolean",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "redacted": {
            "type": "boolean"
        },
        "tags": {
            "type": "array",
            "items": {
                "type": "string",
                "format": "uuid"
            }
        },
        "type": {
            "description": "The type of the new task",
            "example": "MCQ",
            "type": "string",
            "enum": [
                "MCQ",
                "FRQ"
            ]
        },
        "question": {
            "description": "The task question - i.e. what the models should answer",
            "example": "What time is it?",
            "type": "string"
        },
        "answers": {
            "description": "A list of possible answers to be sent to models with the question",
            "type": "array",
            "items": {
                "$ref": "#/components/schemas/MCQAnswer"
            },
            "example": [
                {
                    "text": "half past one",
                    "paraphrases": [
                        "1:30 PM",
                        "13:30"
                    ],
                    "correct": false
                },
                {
                    "text": "Time is an illusion",
                    "correct": false
                },
                {
                    "text": "Now",
                    "correct": true
                }
            ]
        },
        "evaluation_id": {
            "description": "The id of the evaluation that this task is for",
            "type": "string",
            "format": "uuid"
        }
    }
}

Response 201 Created 

{
    "id": "3fd820ae-be67-4b8e-861c-ebcbba814325",
    "task_type": "string",
    "is_task_live": true,
    "modalities": [
        "string"
    ],
    "redacted": true,
    "num_possible_answers": 10.12,
    "evaluation_task_number": 10.12,
    "median_human_completion_seconds": 10.12,
    "median_ai_completion_seconds": 10.12,
    "num_times_human_evaluated": 10.12,
    "num_times_ai_evaluated": 10.12,
    "num_times_humans_answered_correctly": 10.12,
    "num_times_ai_answered_correctly": 10.12,
    "evaluation_id": "d5de6316-0561-4d4a-af52-d29683cd321e",
    "owner_id": "72258a7e-c07e-49eb-a1af-47725539c3b7",
    "tags": [
        {
            "id": "28529464-3cc3-46ab-bad1-6699d0383d05",
            "name": "string"
        }
    ]
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "task_type": {
            "type": "string"
        },
        "is_task_live": {
            "type": "boolean",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "redacted": {
            "type": "boolean"
        },
        "num_possible_answers": {
            "type": "number",
            "format": "int64"
        },
        "evaluation_task_number": {
            "type": "number",
            "format": "int64"
        },
        "median_human_completion_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "median_ai_completion_seconds": {
            "type": "number",
            "format": "double",
            "nullable": true
        },
        "num_times_human_evaluated": {
            "type": "number",
            "format": "int64"
        },
        "num_times_ai_evaluated": {
            "type": "number",
            "format": "int64"
        },
        "num_times_humans_answered_correctly": {
            "type": "number",
            "format": "int64"
        },
        "num_times_ai_answered_correctly": {
            "type": "number",
            "format": "int64"
        },
        "evaluation_id": {
            "type": "string",
            "format": "uuid"
        },
        "owner_id": {
            "type": "string",
            "format": "uuid"
        },
        "tags": {
            "type": "array",
            "items": {
                "$ref": "#/components/schemas/ShallowTag"
            }
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /task

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/Task"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/Task"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /task

Request body

{
    "task_type": "string",
    "is_task_live": true,
    "modalities": [
        "string"
    ],
    "redacted": true,
    "tags": [
        "63a96133-332b-49be-9121-3e852a0a91ae"
    ]
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "task_type": {
            "type": "string"
        },
        "is_task_live": {
            "type": "boolean",
            "nullable": true
        },
        "modalities": {
            "type": "array",
            "items": {
                "type": "string"
            }
        },
        "redacted": {
            "type": "boolean"
        },
        "tags": {
            "type": "array",
            "items": {
                "type": "string",
                "format": "uuid"
            }
        }
    }
}

Response 200 OK 

"Task updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "Task updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

POST /user

Request body

{
    "user_name": "string",
    "full_name": "string",
    "user_image": "string",
    "bio": "string",
    "display_options": null,
    "email_address": "string"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "user_name": {
            "type": "string"
        },
        "full_name": {
            "type": "string",
            "nullable": true
        },
        "user_image": {
            "type": "string",
            "nullable": true
        },
        "bio": {
            "type": "string",
            "nullable": true
        },
        "display_options": {},
        "email_address": {
            "type": "string"
        }
    }
}

Response 201 Created 

{
    "id": "3b5695a6-27b5-4529-92dc-47a89b396837",
    "user_name": "string",
    "full_name": "string",
    "user_image": "string",
    "bio": "string",
    "display_options": null,
    "join_date": "2022-04-13",
    "email_address": "string",
    "subscription_level": "string",
    "alarms": [
        {
            "id": "b9dbf5d5-c13e-4880-990b-9d0396af12c5",
            "name": "They are coming!!",
            "description": "string",
            "metric": "string",
            "threshold": 10.12,
            "public": true,
            "predicted_trigger_date": "2022-04-13",
            "last_trigger_date": "2022-04-13",
            "trigger_cadence": "string",
            "predicted_line_of_best_fit": null,
            "owner_id": "0db4ed95-01f1-4b25-8798-5a36c77e9fe1",
            "evaluation_id": "562a5bbd-548d-4d62-9003-dcdc6810d9e6",
            "owner": "d340bf06-73d4-4cef-9a1c-036bb1f5b71f",
            "evaluation": "ae61dec6-edf7-4ebd-90fb-1582993db1cf",
            "models": [
                "9b5fd31d-5ef9-47b8-bf06-82dcd069d377"
            ]
        }
    ]
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "object",
    "properties": {
        "id": {
            "type": "string",
            "format": "uuid"
        },
        "user_name": {
            "type": "string"
        },
        "full_name": {
            "type": "string",
            "nullable": true
        },
        "user_image": {
            "type": "string",
            "nullable": true
        },
        "bio": {
            "type": "string",
            "nullable": true
        },
        "display_options": {},
        "join_date": {
            "type": "string",
            "format": "date"
        },
        "email_address": {
            "type": "string"
        },
        "subscription_level": {
            "type": "string"
        },
        "alarms": {
            "type": "array",
            "items": {
                "$ref": "#/components/schemas/ShallowAlarm"
            }
        }
    }
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 500 Internal Server Error

Refer to the common response description: Error.

GET /user

Input parameters

Parameter In Type Default Nullable Description
id query string Yes Will return the item with this id, or die trying. When this parameter is provided, then only a single item will be returned

Response 200 OK

Schema of the response body 
{
    "oneOf": [
        {
            "$ref": "#/components/schemas/User"
        },
        {
            "type": "object",
            "properties": {
                "items": {
                    "description": "An array of all the items that were found, but capped at most at `per_page`",
                    "type": "array",
                    "items": {
                        "$ref": "#/components/schemas/User"
                    }
                },
                "count": {
                    "description": "The total number of items found",
                    "type": "number",
                    "format": "int32"
                },
                "per_page": {
                    "description": "The number of items returned per page",
                    "type": "number",
                    "format": "int32"
                },
                "page": {
                    "description": "The number of available pages",
                    "type": "number",
                    "format": "int32"
                }
            }
        }
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

PUT /user

Request body

{
    "user_name": "string",
    "full_name": "string",
    "user_image": "string",
    "bio": "string",
    "display_options": null,
    "email_address": "string"
}
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the request body 
{
    "type": "object",
    "properties": {
        "user_name": {
            "type": "string"
        },
        "full_name": {
            "type": "string",
            "nullable": true
        },
        "user_image": {
            "type": "string",
            "nullable": true
        },
        "bio": {
            "type": "string",
            "nullable": true
        },
        "display_options": {},
        "email_address": {
            "type": "string"
        }
    }
}

Response 200 OK 

"User updated"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "type": "string",
    "enum": [
        "User updated"
    ]
}

Response 401 Unauthorized

Refer to the common response description: Unauthorized.

Response 403 Forbidden

Refer to the common response description: Unauthenticated.

Response 404 Not Found

Refer to the common response description: NotFound.

Response 500 Internal Server Error

Refer to the common response description: Error.

Schemas

Alarm

Name Type
description string| null
evaluation ShallowEvaluation
evaluation_id string(uuid)
id string(uuid)
last_trigger_date string(date)| null
metric string
models Array<ShallowModel>
name string
owner ShallowUser
owner_id string(uuid)
predicted_line_of_best_fit
predicted_trigger_date string(date)| null
public boolean| null
threshold number(double)
trigger_cadence string

ColumnMapping

Name Type
columnType string
paraphraseOf string| null

Evaluation

Name Type
description string| null
id string(uuid)
last_updated string(date-time)
min_questions_to_complete number(int64)| null
modalities Array<string>
name string
num_tasks number(int64)
owner ShallowUser
public boolean
task_types Array<string>

EvaluationEvaluatee

Name Type
cadence string| null
evaluatee_id string(uuid)
evaluation_id string(uuid)
id string(uuid)
model ShallowModel
price number(int64)
status string

EvaluationModelJobs

Name Type
creation_date string(date-time)
evaluation_id string(uuid)
id string(uuid)
job_body
job_description string
job_name string
job_schedule_arn string
minutes_between_evaluations number(int64)
model_id string(uuid)
owner_id string(uuid)
start_date string(date-time)| null

EvaluationSession

Name Type
completed boolean
datetime_completed string(date-time)| null
datetime_started string(date-time)
distribution_of_characters_per_task
distribution_of_seconds_per_task
estimated_session_cost_usd number(double)
evaluatee_id string(uuid)
evaluation_id string(uuid)
failed boolean
id string(uuid)
is_human_being_evaluated boolean
max_characters_per_task number(double)| null
max_seconds_per_task number(double)| null
mean_characters_per_task number(double)| null
mean_seconds_per_task number(double)| null
median_characters_per_task number(double)| null
median_seconds_per_task number(double)| null
min_characters_per_task number(double)| null
min_seconds_per_task number(double)| null
num_answered_correctly number(int64)
num_characters_received_from_endpoint number(int64)
num_characters_sent_to_endpoint number(int64)
num_endpoint_calls number(int64)
num_endpoint_failures number(int64)
num_questions_answered number(int64)
origin string
std_characters_per_task number(double)| null
std_seconds_per_task number(double)| null

MCQAnswer

Name Type
correct boolean
paraphrases Array<string>
text string

Model

Name Type
architecture string| null
best_eval ShallowEvaluationSession
description string| null
endpoint_type string
id string(uuid)
max_characters_per_minute number(int64)
max_request_per_minute number(int64)
modalities Array<string>
name string
num_parameters number(int64)| null
owner ShallowUser
owner_id string(uuid)
picture string| null
public boolean
public_usable boolean
publisher string| null
score number(double)| null
top_example ShallowTask
top_example_id string(uuid)| null
worst_eval ShallowEvaluationSession
worst_example ShallowTask
worst_example_id string(uuid)| null

Response

Name Type
chosen_answer_id string(uuid)| null
correct boolean| null
correctness number(double)| null
creation_date string(date-time)
evaluatee_id string(uuid)
evaluation_session_id string(uuid)
id string(uuid)
parsed_response_text string| null
raw_response_text string| null
raw_task_text string| null
response_time_in_seconds number(double)| null
task_id string(uuid)

ShallowAlarm

Name Type
description string| null
evaluation string(uuid)
evaluation_id string(uuid)
id string(uuid)
last_trigger_date string(date)| null
metric string
models Array<string(uuid)>
name string
owner string(uuid)
owner_id string(uuid)
predicted_line_of_best_fit
predicted_trigger_date string(date)| null
public boolean| null
threshold number(double)
trigger_cadence string

ShallowEvaluation

Name Type
description string| null
id string(uuid)
last_updated string(date-time)
min_questions_to_complete number(int64)| null
modalities Array<string>
name string
num_tasks number(int64)
owner string(uuid)
public boolean
task_types Array<string>

ShallowEvaluationEvaluatee

Name Type
cadence string| null
evaluatee_id string(uuid)
evaluation_id string(uuid)
id string(uuid)
model string(uuid)
price number(int64)
status string

ShallowEvaluationModelJobs

Name Type
creation_date string(date-time)
evaluation_id string(uuid)
id string(uuid)
job_body
job_description string
job_name string
job_schedule_arn string
minutes_between_evaluations number(int64)
model_id string(uuid)
owner_id string(uuid)
start_date string(date-time)| null

ShallowEvaluationSession

Name Type
completed boolean
datetime_completed string(date-time)| null
datetime_started string(date-time)
distribution_of_characters_per_task
distribution_of_seconds_per_task
estimated_session_cost_usd number(double)
evaluatee_id string(uuid)
evaluation_id string(uuid)
failed boolean
id string(uuid)
is_human_being_evaluated boolean
max_characters_per_task number(double)| null
max_seconds_per_task number(double)| null
mean_characters_per_task number(double)| null
mean_seconds_per_task number(double)| null
median_characters_per_task number(double)| null
median_seconds_per_task number(double)| null
min_characters_per_task number(double)| null
min_seconds_per_task number(double)| null
num_answered_correctly number(int64)
num_characters_received_from_endpoint number(int64)
num_characters_sent_to_endpoint number(int64)
num_endpoint_calls number(int64)
num_endpoint_failures number(int64)
num_questions_answered number(int64)
origin string
std_characters_per_task number(double)| null
std_seconds_per_task number(double)| null

ShallowModel

Name Type
architecture string| null
best_eval string(uuid)
description string| null
endpoint_type string
id string(uuid)
max_characters_per_minute number(int64)
max_request_per_minute number(int64)
modalities Array<string>
name string
num_parameters number(int64)| null
owner string(uuid)
owner_id string(uuid)
picture string| null
public boolean
public_usable boolean
publisher string| null
score number(double)| null
top_example string(uuid)
top_example_id string(uuid)| null
worst_eval string(uuid)
worst_example string(uuid)
worst_example_id string(uuid)| null

ShallowResponse

Name Type
chosen_answer_id string(uuid)| null
correct boolean| null
correctness number(double)| null
creation_date string(date-time)
evaluatee_id string(uuid)
evaluation_session_id string(uuid)
id string(uuid)
parsed_response_text string| null
raw_response_text string| null
raw_task_text string| null
response_time_in_seconds number(double)| null
task_id string(uuid)

ShallowTag

Name Type
id string(uuid)
name string

ShallowTask

Name Type
evaluation_id string(uuid)
evaluation_task_number number(int64)
id string(uuid)
is_task_live boolean| null
median_ai_completion_seconds number(double)| null
median_human_completion_seconds number(double)| null
modalities Array<string>
num_possible_answers number(int64)
num_times_ai_answered_correctly number(int64)
num_times_ai_evaluated number(int64)
num_times_human_evaluated number(int64)
num_times_humans_answered_correctly number(int64)
owner_id string(uuid)
redacted boolean
tags Array<string(uuid)>
task_type string

ShallowUser

Name Type
alarms Array<string(uuid)>
bio string| null
display_options
email_address string
full_name string| null
id string(uuid)
join_date string(date)
subscription_level string
user_image string| null
user_name string

Tag

Name Type
id string(uuid)
name string

Task

Name Type
evaluation_id string(uuid)
evaluation_task_number number(int64)
id string(uuid)
is_task_live boolean| null
median_ai_completion_seconds number(double)| null
median_human_completion_seconds number(double)| null
modalities Array<string>
num_possible_answers number(int64)
num_times_ai_answered_correctly number(int64)
num_times_ai_evaluated number(int64)
num_times_human_evaluated number(int64)
num_times_humans_answered_correctly number(int64)
owner_id string(uuid)
redacted boolean
tags Array<ShallowTag>
task_type string

User

Name Type
alarms Array<ShallowAlarm>
bio string| null
display_options
email_address string
full_name string| null
id string(uuid)
join_date string(date)
subscription_level string
user_image string| null
user_name string

Common responses

This section describes common responses that are reused across operations.

Unauthenticated

A valid API token is needed to access this endpoint

"string"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "description": "An error message describing what happened",
    "type": "string"
}

Unauthorized

The provided API token does not have the appropriate permissions to fulfill this request

"string"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "description": "An error message describing what happened",
    "type": "string"
}

NotFound

Could not find this item

"string"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "description": "An error message describing what happened",
    "type": "string"
}

ValidationError

The request has bad data

"string"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "description": "An error message describing what happened",
    "type": "string"
}

Error

A server error

"string"
⚠️ This example has been generated automatically from the schema and it is not accurate. Refer to the schema for more information.

Schema of the response body 
{
    "description": "An error message describing what happened",
    "type": "string"
}

Common parameters

This section describes common parameters that are reused across operations.

apiToken

Name In Type Default Nullable Description
Api-Token header string No