Utils

benchbench.utils.base

benchbench.utils.base.get_combinations(s, k)

Generate all subsets of size k from set s.

Parameters:

• s (list) – List of elements to get combinations from.

• k (int) – Size of each combination.

Returns:

A list of combinations, where each combination is represented as a list.

Return type:

list

benchbench.utils.base.is_int(x)

Check if a string can be converted to an integer.

Parameters:

x (str) – Input string.

Returns:

True if x can be converted to an integer, False otherwise

Return type:

bool

benchbench.utils.base.is_number(s)

Check if a string can be converted to a number.

Parameters:

s (str) – Input string.

Returns:

True if s can be converted to a number, False otherwise

Return type:

bool

benchbench.utils.base.rankdata(a, method='average')

benchbench.utils.metric

benchbench.utils.metric.get_kendall_tau(new_rank, old_rank)

Calculate Kendall’s Tau for two rankings.

Parameters:

• new_rank (np.array) – new ranking

• old_rank (np.array) – old ranking

Returns:

float: Kendall’s Tau float: p-value

Return type:

tuple

benchbench.utils.metric.get_kendall_w(rankings)

Calculate Kendall’s W for a list of rankings.

Parameters:

rankings (list) – a list of rankings

Returns:

Kendall’s W

Return type:

float

benchbench.utils.metric.get_order_diff(new_order, old_order=None)

[Legacy code] Get the difference between two orders.

benchbench.utils.metric.get_order_variance(all_new_order)

[Legacy code] Get the variance of all orders.

benchbench.utils.metric.get_rank_diff(new_rank, old_rank=None)

Get the difference between two ranks.

Parameters:

• new_rank (np.array) – new ranking

• old_rank (np.array) – old ranking

Returns:

Kendall’s Tau float: MRC (max rank change)

Return type:

float

benchbench.utils.metric.get_rank_variance(all_new_rank)

Get the variance of all ranks.

Parameters:

all_new_rank (list) – a list of all rankings

Returns:

w (Kendall’s W) float: max_MRC (the max MRC over every pair of rankings)

Return type:

float

benchbench.utils.metric.order2rank(order)

[Legacy code] Convert an order to a rank.

benchbench.utils.metric.rank2order(rank)

[Legacy code] Convert a rank to an order.

benchbench.utils.win_rate

class benchbench.utils.win_rate.WinningRate(data, cols)

Bases: object

get_winning_rate(model_indices=None)

Get the winning rate of the selected models.

Parameters:

model_indices (list) – Indices of the selected models.

Returns:

The winning rate.

Return type:

float