utils
Attributes
Classes
A class for sampling text lengths within a given range. |
Functions
|
Determines the device for model computation. |
|
Fetches NVIDIA GPU details using the nvidia-smi command. |
|
Saves results to a JSON file. |
|
Cleans and trims decoded outputs to the last complete sentence. |
Retrieves harmless prompts from the Anthropic dataset. |
|
Retrieves prompts from the IMDB dataset, filtered by length. |
|
|
Calculates humor probabilities and raw scores. |
|
Calculates positive sentiment probabilities and scores. |
|
Calculates harmlessness probabilities and scores. |
|
Calculates probabilities and raw scores indicating the harmlessness of a response. |
|
Calculates probabilities and raw scores indicating the helpfulness of a response. |
|
Calculates the unique fraction of n-grams within a sentence. |
|
Calculates diversity score by computing unique n-grams for n=2,3,4. |
|
Calculates log perplexity for a list of sentences. |
|
Calculates coherence between prompts and continuations using cosine similarity. |
|
Calculates reward scores based on the specified value (e.g., humor, coherence). |
Converts a PPO-trained model name to a valid Hugging Face model path format. |
|
|
Fetches a model and tokenizer based on the model name. |
Module Contents
- utils.TASK_NAME = 'conversation'
- utils.ALL_SUPPORTED_VALUES = ['humor', 'gpt2-helpful', 'gpt2-harmless', 'diversity', 'coherence', 'perplexity']
- utils.get_device() str
Determines the device for model computation.
- Returns:
Device type, either “cuda” if available, else “cpu”.
- Return type:
str
- utils.get_nvidia_smi_info() str | None
Fetches NVIDIA GPU details using the nvidia-smi command.
- Returns:
Output from nvidia-smi command if successful, None otherwise.
- Return type:
str or None
- utils.devices
- utils.save_results_to_json(results: list, file_path: str) None
Saves results to a JSON file.
- Parameters:
results (list) – List of dictionaries containing prompt and generated text.
file_path (str) – Path to save the JSON file.
- utils.clean_and_trim_to_last_sentence(prompts: List[str], decoded_outputs: List[str]) List[str]
Cleans and trims decoded outputs to the last complete sentence.
- Parameters:
prompts (List[str]) – List of input prompts.
decoded_outputs (List[str]) – List of decoded model outputs.
- Returns:
Cleaned and trimmed outputs.
- Return type:
List[str]
- class utils.LengthSampler(min_length, max_length)
A class for sampling text lengths within a given range.
- min_length
- max_length
- __call__()
Generates a random length within the specified range.
- Returns:
Random length within the min and max range.
- Return type:
int
- trim_to_word_boundary(text, length)
Trims text to the last word boundary within a given length.
- Parameters:
text (str) – Text to be trimmed.
length (int) – Maximum character length for trimming.
- Returns:
Text trimmed to the last word boundary.
- Return type:
str
- utils.get_prompts_from_Anthropic_harmless()
Retrieves harmless prompts from the Anthropic dataset. Importantly, the generation is non-random everytime being called.
- Returns:
List of harmless prompts.
- Return type:
List[str]
- utils.get_prompts_from_imdb()
Retrieves prompts from the IMDB dataset, filtered by length. Importantly, the generation is non-random everytime being called.
- Returns:
List of filtered IMDB prompts.
- Return type:
List[str]
- utils.cal_humor_probabilities(sentences, model, tokenizer)
Calculates humor probabilities and raw scores.
- Parameters:
sentences (List[str]) – List of sentences, whose size/batchsize is limited by memory
model – Model for humor classification.
tokenizer – Tokenizer for input processing.
- Returns:
Humor probabilities and raw scores.
- Return type:
Tuple[List[float], List[float]]
- utils.cal_positive_sentiment(sentences, model, tokenizer)
Calculates positive sentiment probabilities and scores.
- Parameters:
sentences (List[str]) – List of sentences.
model – Model for sentiment classification.
tokenizer – Tokenizer for input processing.
- Returns:
Positive probabilities and scores.
- Return type:
Tuple[List[float], List[float]]
- utils.cal_harmless_probabilities(sentences: List[str], model, tokenizer) Tuple[List[float], List[float]]
Calculates harmlessness probabilities and scores.
- Parameters:
sentences (List[str]) – List of sentences.
model – Model for harmlessness classification.
tokenizer – Tokenizer for input processing.
- Returns:
Harmless probabilities and scores.
- Return type:
Tuple[List[float], List[float]]
- utils.cal_gpt2_harmless_probabilities(prompts: List[str], continuations: List[str], model, tokenizer) Tuple[List[float], List[float]]
Calculates probabilities and raw scores indicating the harmlessness of a response.
- Parameters:
prompts (List[str]) – List of initial prompt strings.
continuations (List[str]) – List of continuation (response) strings.
model – Model for evaluating harmlessness.
tokenizer – Tokenizer for processing the inputs.
- Returns:
Harmlessness probabilities and raw scores.
- Return type:
Tuple[List[float], List[float]]
- utils.cal_gpt2_helpful_probabilities(prompts: List[str], continuations: List[str], model, tokenizer) Tuple[List[float], List[float]]
Calculates probabilities and raw scores indicating the helpfulness of a response.
- Parameters:
prompts (List[str]) – List of prompt strings.
continuations (List[str]) – List of continuation (response) strings.
model – Model for evaluating helpfulness.
tokenizer – Tokenizer for processing the inputs.
- Returns:
Helpfulness probabilities and raw scores.
- Return type:
Tuple[List[float], List[float]]
- utils.compute_rep_n(sentence: str, n: int) float
Calculates the unique fraction of n-grams within a sentence. This function is a subroutine of cal_diversity
- Parameters:
sentence (str) – Sentence to analyze.
n (int) – Length of the n-grams to compute.
- Returns:
Fraction of unique n-grams in the sentence.
- Return type:
float
- utils.cal_diversity(sentence: str) float
Calculates diversity score by computing unique n-grams for n=2,3,4.
- Parameters:
sentence (str) – Sentence to analyze.
- Returns:
Diversity score, product of unique n-grams for each n in (2, 3, 4).
- Return type:
float
- utils.cal_log_perplexity(sentences: List[str], model, tokenizer) List[float]
Calculates log perplexity for a list of sentences.
- Parameters:
sentences (List[str]) – List of sentences.
model – Model to evaluate perplexity.
tokenizer – Tokenizer for processing the inputs.
- Returns:
List of log perplexity values for each sentence.
- Return type:
List[float]
- utils.cal_coherence(prompts: List[str], continuations: List[str], model, tokenizer) List[float]
Calculates coherence between prompts and continuations using cosine similarity.
- Parameters:
prompts (List[str]) – List of prompt sentences.
continuations (List[str]) – List of continuation sentences. Equal length as prompts as each prompt corresponds to each sentence.
model – Model for sentence embedding.
tokenizer – Tokenizer for processing the inputs.
- Returns:
List of cosine similarity scores for each prompt-continuation pair.
- Return type:
List[float]
- utils.get_reward(sentences: List[str], value: str, model=None, tokenizer=None, prompts: List[str] = None, use_score: bool = True) List[float] | None
Calculates reward scores based on the specified value (e.g., humor, coherence).
- Parameters:
sentences (List[str]) – List of sentences for evaluation. Each consists of a prompt and a continuation.
value (str) – Type of reward (e.g., “humor”, “positive”, etc.).
model – Model used for evaluation, if applicable.
tokenizer – Tokenizer used for model input processing.
prompts (List[str], optional) – List of prompts if needed for evaluation. Required to compute the gpt2_harmless, gpt2_helpful, coherence.
use_score (bool, optional) – If True, returns raw scores; otherwise, probabilities.
- Returns:
List of reward scores or probabilities, or None if the value is invalid.
- Return type:
Union[List[float], None]
Example
>>> rewards = get_reward(sentences, "humor") >>> print("Average Reward:", np.mean(rewards))
- utils.convert_ppo_modelname_to_huggingface_valid(ppo_model_name: str) str
Converts a PPO-trained model name to a valid Hugging Face model path format.
- Parameters:
ppo_model_name (str) – Model name to be converted.
- Returns:
Converted model name.
- Return type:
str
- utils.get_model_and_tokenizer(model_name: str)
Fetches a model and tokenizer based on the model name.
- Parameters:
model_name (str) – Name of the model.
- Returns:
The specified model and tokenizer.
- Return type:
Tuple[transformers.PreTrainedModel, transformers.PreTrainedTokenizer]