Generation configs, results and errors

Generation Configs

GenConf dataclass

Model generation configuration, used in Model.gen() and variants.

max_tokens class-attribute instance-attribute

max_tokens = 0

Maximum output token length. Special value of 0 means all available context length, special values between -1 and -100 mean a -percentage of ctx_len. In some providers, a value of 0 also signals that max_tokens is not used/sent. For example -20 allows output up to 20% of ctx_len.

stop class-attribute instance-attribute

stop = field(default_factory=list)

List of generation stop text sequences

temperature class-attribute instance-attribute

temperature = 0.0

Generation temperature. Use 0 to always pick the most probable output, without random sampling. Larger positive values will produce more random outputs.

top_p class-attribute instance-attribute

top_p = 0.9

Nucleus sampling top_p value. Only applies if temperature > 0.

format class-attribute instance-attribute

format = 'text'

Output format: "text" or "json". For JSON output, text is validated as in json.loads(). Thread msgs must explicitly request JSON output or a warning will be emitted if string json not present (this is automatically done in Model.json() and related calls).

json_schema class-attribute instance-attribute

json_schema = None

A JSON schema to validate the JSON output. Thread msgs must list the JSON schema and request its use; must also set the format to "json".

special class-attribute instance-attribute

special = None

Special model or provider-specific generation arguments. Args in the base dict are included unconditionally for a model, while args in sub-keys with the model's provider name are only used for models from that provider, for example "openai": {...} values are only used in OpenAI models.

__call__

__call__(**kwargs)

Return a copy of the current GenConf updated with values in kwargs. Doesn't modify object. Key 'special' is updated element-wise.

Parameters:

Name	Type	Description	Default
**kwargs	Any	update settings of the same names in the returned copy.	{}

Raises:

Type	Description
KeyError	If key does not exist.

Returns:

Type	Description
Self	A copy of the current object with kwargs values updated. Doesn't modify object.

Source code in sibila/gen.py

def __call__(self,
             **kwargs: Any) -> Self:
    """Return a copy of the current GenConf updated with values in kwargs. Doesn't modify object.
    Key 'special' is updated element-wise.

    Args:
        **kwargs: update settings of the same names in the returned copy.

    Raises:
        KeyError: If key does not exist.

    Returns:
        A copy of the current object with kwargs values updated. Doesn't modify object.
    """

    ret = deepcopy(self)

    for k,v in kwargs.items():
        if not hasattr(ret, k):
            raise KeyError(f"No such key '{k}'")
        if k == "special":
            if ret.special is None:
                ret.special = {}
            if v is None:
                v = {}
            ret.special.update(v)
            if not ret.special:
                ret.special = None
        else:
            setattr(ret, k,v)

    return ret

clone

clone()

Return a deep copy of this configuration.

Source code in sibila/gen.py

def clone(self) -> Self:
    """Return a deep copy of this configuration."""
    return deepcopy(self)

as_dict

as_dict()

Return GenConf as a dict.

Source code in sibila/gen.py

def as_dict(self) -> dict:
    """Return GenConf as a dict."""
    return asdict(self)

from_dict staticmethod

from_dict(dic)

Source code in sibila/gen.py

@staticmethod
def from_dict(dic: dict) -> Any: # Any = GenConf
    return GenConf(**dic)

resolve_max_tokens

resolve_max_tokens(ctx_len, max_tokens_limit=None)

Calculate actual max_tokens value for cases where it's zero or a percentage of model's ctx_len)

Parameters:

Name	Type	Description	Default
ctx_len	int	Model's context length.	required
max_tokens_limit	Optional[int]	Optional model's limit for max_tokens. Defaults to None.	None

Returns:

Type	Description
int	An actual model maximum number of output tokens.

Source code in sibila/gen.py

def resolve_max_tokens(self,
                       ctx_len: int,
                       max_tokens_limit: Optional[int] = None) -> int:
    """Calculate actual max_tokens value for cases where it's zero or a percentage of model's ctx_len)

    Args:
        ctx_len: Model's context length.
        max_tokens_limit: Optional model's limit for max_tokens. Defaults to None.

    Returns:
        An actual model maximum number of output tokens.
    """

    max_tokens = self.max_tokens
    if max_tokens <= 0:
        if max_tokens == 0:
            max_tokens = ctx_len
        else:
            max_tokens = min(-max_tokens, 100)
            max_tokens = int(max_tokens / 100.0 * ctx_len)
            max_tokens = max(1,max_tokens)

    if max_tokens_limit is not None:
        max_tokens = min(max_tokens, max_tokens_limit)

    return max_tokens

resolve_special

resolve_special(provider=None)

Compiles settings from the 'special' field, for model and provider.

Parameters:

Name	Type	Description	Default
provider	Optional[str]	If set will include any 'special' settings specified for that provider, inside a key named after the provider. If not given, only base keys are added.	None

Returns:

Type	Description
dict	description

Source code in sibila/gen.py

def resolve_special(self, 
                    provider: Optional[str] = None) -> dict:
    """Compiles settings from the 'special' field, for model and provider.

    Args:
        provider: If set will include any 'special' settings specified for that provider, inside a key named after the provider. If not given, only base keys are added.

    Returns:
        _description_
    """

    if self.special is None:
        return {}

    from .models import Models

    out = {}
    for k,v in self.special.items():
        if k == provider: # provider-specific
            if not isinstance(v,dict):
                raise ValueError(f"Config 'special' for provider '{provider}' must be a dict")
            out.update(v)
        else: # common args
            if isinstance(v,dict) and k in Models.ALL_PROVIDER_NAMES: # skip other provider entries
                continue
            out[k] = v
    return out

JSchemaConf dataclass

Configuration for JSON schema massaging and validation.

resolve_refs class-attribute instance-attribute

resolve_refs = True

Set for $ref references to be resolved and replaced with actual definition.

collapse_single_combines class-attribute instance-attribute

collapse_single_combines = True

Any single-valued "oneOf"/"anyOf" is replaced with the actual value.

description_from_title class-attribute instance-attribute

description_from_title = 0

If a value doesn't have a description entry, make one from its title or name.

• 0: don't make description from name
• 1: copy title or name to description
• 2: 1: + capitalize first letter and convert _ to space: class_label -> "class label".

force_all_required class-attribute instance-attribute

force_all_required = False

Force all entries in an object to be required (except removed defaults if remove_with_default=True).

remove_with_default class-attribute instance-attribute

remove_with_default = False

Delete any values that have a "default" annotation.

default_to_last class-attribute instance-attribute

default_to_last = True

Move any default value entry into the last position of properties dict.

additional_allowed_root_keys class-attribute instance-attribute

additional_allowed_root_keys = field(default_factory=list)

By default only the following properties are allowed in schema's root: description, properties, type, required, additionalProperties, allOf, anyOf, oneOf, not Add to this list to allow additional root properties.

pydantic_strict_validation class-attribute instance-attribute

pydantic_strict_validation = None

Validate JSON values in a strict manner or not. None means validate individually per each value in the obj. (for example in pydantic with: Field(strict=True)).

clone

clone()

Return a copy of this configuration.

Source code in sibila/json_schema.py

def clone(self):
    """Return a copy of this configuration."""
    return copy(self)

Results

GenRes

Model generation result.

OK_STOP class-attribute instance-attribute

OK_STOP = 1

Generation complete without errors.

OK_LENGTH class-attribute instance-attribute

OK_LENGTH = 0

Generation stopped due to reaching max_tokens.

ERROR_JSON class-attribute instance-attribute

ERROR_JSON = -1

Invalid JSON: this is often due to the model returning OK_LENGTH (finished due to max_tokens reached), which cuts off the JSON text.

ERROR_JSON_SCHEMA_VAL class-attribute instance-attribute

ERROR_JSON_SCHEMA_VAL = -2

Failed JSON schema validation.

ERROR_JSON_SCHEMA_ERROR class-attribute instance-attribute

ERROR_JSON_SCHEMA_ERROR = -2

JSON schema itself is not valid.

ERROR_MODEL class-attribute instance-attribute

ERROR_MODEL = -3

Other model internal error.

from_finish_reason staticmethod

from_finish_reason(finish)

Convert a ChatCompletion finish result into a GenRes.

Parameters:

Name	Type	Description	Default
finish	str	ChatCompletion finish result.	required

Returns:

Type	Description
Any	A GenRes result.

Source code in sibila/gen.py

@staticmethod
def from_finish_reason(finish: str) -> Any: # Any=GenRes
    """Convert a ChatCompletion finish result into a GenRes.

    Args:
        finish: ChatCompletion finish result.

    Returns:
        A GenRes result.
    """
    if finish == 'stop':
        return GenRes.OK_STOP
    elif finish == 'length':
        return GenRes.OK_LENGTH
    elif finish == '!json':
        return GenRes.ERROR_JSON
    elif finish == '!json_schema_val':
        return GenRes.ERROR_JSON_SCHEMA_VAL
    elif finish == '!json_schema_error':
        return GenRes.ERROR_JSON_SCHEMA_ERROR
    else:
        return GenRes.ERROR_MODEL

as_text staticmethod

as_text(res)

Returns a friendlier description of the result.

Parameters:

Name	Type	Description	Default
res	Any	Model output result.	required

Raises:

Type	Description
ValueError	If unknown GenRes.

Returns:

Type	Description
str	A friendlier description of the GenRes.

Source code in sibila/gen.py

@staticmethod
def as_text(res: Any) -> str: # Any=GenRes
    """Returns a friendlier description of the result.

    Args:
        res: Model output result.

    Raises:
        ValueError: If unknown GenRes.

    Returns:
        A friendlier description of the GenRes.
    """

    if res == GenRes.OK_STOP:
        return "Stop"
    elif res == GenRes.OK_LENGTH:
        return "Length (output cut)"
    elif res == GenRes.ERROR_JSON:
        return "JSON decoding error"

    elif res == GenRes.ERROR_JSON_SCHEMA_VAL:
        return "JSON SCHEMA validation error"
    elif res == GenRes.ERROR_JSON_SCHEMA_ERROR:
        return "Error in JSON SCHEMA"

    elif res == GenRes.ERROR_MODEL:
        return "Model internal error"
    else:
        raise ValueError("Bad/unknow GenRes")

Errors

GenError

GenError(out)

Model generation exception, raised when the model was unable to return a response.

An error has happened during model generation.

Parameters:

Name	Type	Description	Default
out	GenOut	Model output	required

Source code in sibila/gen.py

def __init__(self, 
             out: GenOut):
    """An error has happened during model generation.

    Args:
        out: Model output
    """

    assert out.res != GenRes.OK_STOP, "OK_STOP is not an error"      

    super().__init__()

    self.res = out.res
    self.text = out.text
    self.dic = out.dic
    self.value = out.value

raise_if_error staticmethod

raise_if_error(out, ok_length_is_error)

Raise an exception if the model returned an error

Parameters:

Name	Type	Description	Default
out	GenOut	Model returned info.	required
ok_length_is_error	bool	Should a result of GenRes.OK_LENGTH be considered an error?	required

Raises:

Type	Description
GenError	If an error was returned by model.

Source code in sibila/gen.py

@staticmethod
def raise_if_error(out: GenOut,
                   ok_length_is_error: bool):
    """Raise an exception if the model returned an error

    Args:
        out: Model returned info.
        ok_length_is_error: Should a result of GenRes.OK_LENGTH be considered an error?

    Raises:
        GenError: If an error was returned by model.
    """

    if out.res != GenRes.OK_STOP:
        if out.res == GenRes.OK_LENGTH and not ok_length_is_error:
            return # set ok_length_is_error to ignore this error

        raise GenError(out)

GenOut dataclass

Model output, returned by gen_extract(), gen_json() and other model calls that don't raise exceptions.

res instance-attribute

res

Result of model generation.

text instance-attribute

text

Text generated by model.

dic class-attribute instance-attribute

dic = None

Python dictionary, output by the structured calls like gen_json().

value class-attribute instance-attribute

value = None

Initialized instance value, dataclass or Pydantic BaseModel object, as returned in calls like extract().

as_dict

as_dict()

Return GenOut as a dict.

Source code in sibila/gen.py

def as_dict(self):
    """Return GenOut as a dict."""
    return asdict(self)

__str__

__str__()

Source code in sibila/gen.py

def __str__(self):
    out = f"Error={self.res.as_text(self.res)} text=█{self.text}█"
    if self.dic is not None:
        out += f" dic={self.dic}"
    if self.value is not None:
        out += f" value={self.value}"
    return out