Welcome to folktexts’ documentation!

The folktexts package enables you to benchmark and evaluate LLMs as risk scores.

We encode unrealizable tabular prediction tasks as natural-language text, and prompt LLMs for the probability of target variable being true. The correct solutions for each task often require expressing uncertainty, as the target variable is not uniquely determined by the input features.

Folktexts is compatible with any huggingface transformer model.

Five tabular data tasks are provided out-of-the-box, using the American Community Survey as a data source: ACSIncome, ACSMobility, ACSTravelTime, ACSEmployment, and ACSPublicCoverage. These tasks follow the same name, feature columns, and target columns as those put forth by Ding et al. (2021) in the folktables python package.

Full code available on the GitHub repository, including various jupyter notebook examples .

Check out the following sub-pages:

• Readme file
• API reference
• Example notebooks

Indices

• Index

• Module Index