prompting.validators.criteria

Module Contents

Classes

TaskCriterion	Abstract base class for defining task-specific evaluation criteria.
TextLengthUnitEnum	Create a collection of name/value pairs.
MatchLengthCriteria	Abstract base class for defining task-specific evaluation criteria.
ContentMatchTypeEnum	Create a collection of name/value pairs.
MatchContentCriteria	Abstract base class for defining task-specific evaluation criteria.
SimpleResponseLayoutCriteria	Abstract base class for defining task-specific evaluation criteria.
LayoutMatchTypeEnum	Create a collection of name/value pairs.
MatchLayoutCriteria	Abstract base class for defining task-specific evaluation criteria.

class prompting.validators.criteria.TaskCriterion

Bases: abc.ABC

Abstract base class for defining task-specific evaluation criteria.

text

Text of the criterion to be added to the prompt.

Type:

str

penalty

Penalty value associated with the criterion.

Type:

float

Returns:

Tensor containing the penalty values for each response.

Return type:

torch.FloatTensor

text: str

penalty: float

abstract evaluate(completions)

Parameters:

completions (List[str]) –

Return type:

torch.FloatTensor

abstract compose_text()

Return type:

str

class prompting.validators.criteria.TextLengthUnitEnum(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

• attribute access:

  >>> Color.RED
  <Color.RED: 1>
  

• value lookup:

  >>> Color(1)
  <Color.RED: 1>
  

• name lookup:

  >>> Color['RED']
  <Color.RED: 1>
  

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3

>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

CHARACTERS = 'characters'

WORDS = 'words'

SENTENCES = 'sentences'

PARAGRAPHS = 'paragraphs'

class prompting.validators.criteria.MatchLengthCriteria

Bases: TaskCriterion

Abstract base class for defining task-specific evaluation criteria.

text

Text of the criterion to be added to the prompt.

Type:

str

penalty

Penalty value associated with the criterion.

Type:

float

Returns:

Tensor containing the penalty values for each response.

Return type:

torch.FloatTensor

text: str = 'Your response must have {target_length} {unit}.'

penalty: float = 0.1

target_length: int = 100

unit: TextLengthUnitEnum

_count_sentences(text)

_get_completion_length(response)

Parameters:

response (str) –

Return type:

int

evaluate(completions)

Parameters:

completions (List[str]) –

Return type:

torch.FloatTensor

compose_text()

Return type:

str

class prompting.validators.criteria.ContentMatchTypeEnum(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

• attribute access:

  >>> Color.RED
  <Color.RED: 1>
  

• value lookup:

  >>> Color(1)
  <Color.RED: 1>
  

• name lookup:

  >>> Color['RED']
  <Color.RED: 1>
  

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3

>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

STARTS_WITH = 'starts with'

ENDS_WITH = 'ends with'

INCLUDES = 'includes'

class prompting.validators.criteria.MatchContentCriteria

Bases: TaskCriterion

Abstract base class for defining task-specific evaluation criteria.

text

Text of the criterion to be added to the prompt.

Type:

str

penalty

Penalty value associated with the criterion.

Type:

float

Returns:

Tensor containing the penalty values for each response.

Return type:

torch.FloatTensor

default_text: str = 'Your response should {match_type} the following words: {words}.'

text: str

penalty: float = 0.1

n_words: int = 3

words_array: List[str]

contentMatchType: ContentMatchTypeEnum

sampled_words: List[str]

negate_match: bool = False

__post_init__()

_get_regex_pattern()

evaluate(completions)

Parameters:

completions (List[str]) –

Return type:

torch.FloatTensor

compose_text()

Return type:

str

class prompting.validators.criteria.SimpleResponseLayoutCriteria

Bases: TaskCriterion

Abstract base class for defining task-specific evaluation criteria.

text

Text of the criterion to be added to the prompt.

Type:

str

penalty

Penalty value associated with the criterion.

Type:

float

Returns:

Tensor containing the penalty values for each response.

Return type:

torch.FloatTensor

penalty: float = 0.1

text: str = 'Your response should not contain any bullet points or numbered lists.'

evaluate(completions)

Parameters:

completions (List[str]) –

Return type:

torch.FloatTensor

compose_text()

Return type:

str

class prompting.validators.criteria.LayoutMatchTypeEnum(*args, **kwds)

Bases: enum.Enum

Create a collection of name/value pairs.

Example enumeration:

>>> class Color(Enum):
...     RED = 1
...     BLUE = 2
...     GREEN = 3

Access them by:

• attribute access:

  >>> Color.RED
  <Color.RED: 1>
  

• value lookup:

  >>> Color(1)
  <Color.RED: 1>
  

• name lookup:

  >>> Color['RED']
  <Color.RED: 1>
  

Enumerations can be iterated over, and know how many members they have:

>>> len(Color)
3

>>> list(Color)
[<Color.RED: 1>, <Color.BLUE: 2>, <Color.GREEN: 3>]

Methods can be added to enumerations, and members can have their own attributes – see the documentation for details.

UNORDERED_LIST = 'unordered list'

NUMBERED_LIST = 'numbered list'

class prompting.validators.criteria.MatchLayoutCriteria

Bases: TaskCriterion

Abstract base class for defining task-specific evaluation criteria.

text

Text of the criterion to be added to the prompt.

Type:

str

penalty

Penalty value associated with the criterion.

Type:

float

Returns:

Tensor containing the penalty values for each response.

Return type:

torch.FloatTensor

layout_type: LayoutMatchTypeEnum

penalty: float = 0.1

text: str = 'Your response should be ordered in format of {layout_type}.'

evaluate(completions)

Parameters:

completions (List[str]) –

Return type:

torch.FloatTensor

compose_text()

Return type:

str