Postprocessing frontier (Pareto curve)

Use error_parity.pareto_curve.compute_postprocessing_curve() to compute the fairness–performance frontier across tolerances.

API

error_parity.pareto_curve.compute_postprocessing_curve(model, fit_data, eval_data, fairness_constraint='equalized_odds', l_p_norm=inf, bootstrap=True, tolerance_ticks=array([0., 0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.1, 0.11, 0.12, 0.13, 0.14, 0.15, 0.16, 0.17, 0.18, 0.19, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]), tolerance_tick_step=None, predict_method='predict_proba', n_jobs=None, **kwargs)

Computes the fairness and performance of the given classifier after adjusting (postprocessing) for varying levels of fairness tolerance.

Parameters:

• model (object) – The model to use.

• fit_data (tuple) – Data triplet to use to fit postprocessing intervention, (X, Y, S), respectively containing the features, labels, and sensitive attribute.

• eval_data (tuple or dict[tuple]) – Data triplet to use to evaluate postprocessing intervention on (same format as fit_data), or a dictionary of <data_name>-><data_triplet> containing multiple datasets to evaluate on.

• fairness_constraint (str, optional) – The fairness constraint to use , by default “equalized_odds”.

• l_p_norm (int, optional) – The norm to use when computing the fairness constraint, by default np.inf. Note: only compatible with the “equalized odds” constraint.

• bootstrap (bool, optional) – Whether to compute uncertainty estimates via bootstrapping, by default False.

• tolerance_ticks (list, optional) – List of constraint tolerances to use when computing adjustment curve. By default will use higher granularity/precision for lower levels of disparity, and lower granularity for higher levels of disparity. Should correspond to a sorted list of values between 0 and 1. Will be ignored if tolerance_tick_step is provided.

• tolerance_tick_step (float, optional) – Distance between constraint tolerances in the adjustment curve. Will override tolerance_ticks if provided!

• predict_method (str, optional) – Which method to call to obtain predictions out of the given model. Use predict_method=”__call__” for a callable predictor, or the default predict_method=”predict_proba” for a predictor with sklearn interface.

• n_jobs (int, optional) – Number of parallel jobs to use, if omitted will use os.cpu_count()-1.

Returns:

postproc_results_df – A DataFrame containing the results, one row per tolerance tick.

Return type:

pd.DataFrame

Interpretation

• The returned pandas.DataFrame has one row per tolerance tick and columns for each metric and dataset split.

• Use error_parity.plotting.plot_postprocessing_frontier() to visualize the envelope of the frontier and optional bootstrap confidence intervals.

Related utilities

error_parity.pareto_curve.fit_and_evaluate_postprocessing(postproc_template, tolerance, fit_data, eval_data, seed=42, y_fit_pred_scores=None, bootstrap=True, **bootstrap_kwargs)

Fit and evaluate a postprocessing intervention on the given predictor.

Parameters:

• postproc_template (RelaxedThresholdOptimizer) – An object that serves as the template to copy when creating the postprocessing optimizer.

• tolerance (float) – The tolerance (or slack) for fairness constraint fulfillment. This value will override the tolerance attribute of the postproc_template object.

• fit_data (tuple) – The data used to fit postprocessing.

• eval_data (tuple or dict[tuple]) – The data or sequence of data to evaluate postprocessing on. If a tuple is provided, will call it “eval” data in the returned results dictionary; if a dict is provided, will assume {<key_1>: <data_1>, …}.

• seed (int, optional) – The random seed, by default 42

• y_fit_pred_scores (np.ndarray, optional) – The pre-computed predicted scores for the fit_data; if provided, will avoid re-computing these predictions for each function call.

• bootstrap (bool, optional) – Whether to use bootstrapping when computing metric results for postprocessing, by default True.

• bootstrap_kwargs (dict, optional) – Any extra arguments to pass on to the bootstrapping function, by default None.

Returns:

results – A dictionary of results, whose keys are the data type, and values the metric values obtained by postprocessing on that data type.

For example: >>> { >>> “validation”: {“accuracy”: 0.7, “…”: “…”}, >>> “test”: {“accuracy”: 0.65, “…”: “…”}, >>> }

Return type:

dict[str, dict]

error_parity.pareto_curve.get_envelope_of_postprocessing_frontier(postproc_results_df, perf_col='accuracy_mean_test', disp_col='equalized_odds_diff_mean_test', constant_clf_perf=0.5, constant_clf_disp=0.0)

Computes points in envelope of the given postprocessing frontier results.

Parameters:

• postproc_results_df (pd.DataFrame) – The postprocessing frontier results DF.

• perf_col (str, optional) – Name of the column containing performance results, by default “accuracy_mean_test”

• disp_col (str, optional) – Name of column containing disparity results, by default “equalized_odds_diff_mean_test”

• constant_clf_perf (float, optional) – The performance of a dummy constant classifier (in the same metric as perf_col), by default 0.5.

• constant_clf_disp (float, optional) – The disparity of a dummy constant classifier (in the same metric as disp_col), by default 0.0; assumes a constant classifier fulfills fairness!

Returns:

A 2-D array containing the points in the convex hull of the Pareto curve.

Return type:

np.ndarray

error_parity.pareto_curve.compute_inner_and_outer_adjustment_ci(postproc_results_df, perf_metric, disp_metric, data_type='test', constant_clf_perf=None)

Computes the interior/inner and exterior/outer adjustment curves, corresponding to the confidence intervals (by default 95% c.i.).

Returns:

postproc_results_df – A tuple containing (xticks, inner_yticks, outer_yticks).

Return type:

tuple[np.array, np.array, np.array]

Parameters:

• perf_metric (str)

• disp_metric (str)

• data_type (str)

• constant_clf_perf (float | None)