SWI-Prolog -- library(aggregate): Aggregation operators on backtrackable predicates

A.1 library(aggregate): Aggregation operators on backtrackable predicates

Compatibility

Quintus, SICStus 4. The forall/2 is a SWI-Prolog built-in and term_variables/3 is a SWI-Prolog built-in with different semantics.

To be done

- Analysing the aggregation template and compiling a predicate for the list aggregation can be done at compile time.
- aggregate_all/3 can be rewritten to run in constant space using non-backtrackable assignment on a term.

This library provides aggregating operators over the solutions of a predicate. The operations are a generalisation of the bagof/3, setof/3 and findall/3 built-in predicates. Aggregations that can be computed incrementally avoid findall/3 and run in constant memory. The defined aggregation operations are counting, computing the sum, minimum, maximum, a bag of solutions and a set of solutions. We first give a simple example, computing the country with the smallest area:

smallest_country(Name, Area) :- aggregate(min(A, N), country(N, A), min(Area, Name)).

There are four aggregation predicates (aggregate/3, aggregate/4, aggregate_all/3 and aggregate/4), distinguished on two properties.

aggregate vs. aggregate_all

The aggregate predicates use setof/3 (aggregate/4) or bagof/3 (aggregate/3), dealing with existential qualified variables (Var^Goal) and providing multiple solutions for the remaining free variables in Goal. The aggregate_all/3 predicate uses findall/3, implicitly qualifying all free variables and providing exactly one solution, while aggregate_all/4 uses sort/2 over solutions that Discriminator (see below) generated using findall/3.

The Discriminator argument

The versions with 4 arguments deduplicate redundant solutions of Goal. Solutions for which both the template variables and Discriminator are identical will be treated as one solution. For example, if we wish to compute the total population of all countries, and for some reason country(belgium, 11000000) may succeed twice, we can use the following to avoid counting the population of Belgium twice:

    aggregate(sum(P), Name, country(Name, P), Total)

All aggregation predicates support the following operators below in Template. In addition, they allow for an arbitrary named compound term, where each of the arguments is a term from the list below. For example, the term r(min(X), max(X)) computes both the minimum and maximum binding for X.

count

Count number of solutions. Same as sum(1).

sum(Expr)

Sum of Expr for all solutions.

min(Expr)

Minimum of Expr for all solutions.

min(Expr, Witness)

A term min(Min, Witness), where Min is the minimal version of Expr over all solutions, and Witness is any other template applied to solutions that produced Min. If multiple solutions provide the same minimum, Witness corresponds to the first solution.

max(Expr)

Maximum of Expr for all solutions.

max(Expr, Witness)

As min(Expr, Witness), but producing the maximum result.

set(X)

An ordered set with all solutions for X.

bag(X)

A list of all solutions for X.

Acknowledgements

The development of this library was sponsored by SecuritEase, http://www.securitease.com

[nondet]aggregate(+Template, :Goal, -Result)

Aggregate bindings in Goal according to Template. The aggregate/3 version performs bagof/3 on Goal.

[nondet]aggregate(+Template, +Discriminator, :Goal, -Result)

Aggregate bindings in Goal according to Template. The aggregate/4 version performs setof/3 on Goal.

[semidet]aggregate_all(+Template, :Goal, -Result)

Aggregate bindings in Goal according to Template. The aggregate_all/3 version performs findall/3 on Goal. Note that this predicate fails if Template contains one or more of min(X), max(X), min(X,Witness) or max(X,Witness) and Goal has no solutions, i.e., the minimum and maximum of an empty set is undefined.

The Template values count, sum(X), max(X), min(X), max(X,W) and min(X,W) are processed incrementally rather than using findall/3 and run in constant memory.

[semidet]aggregate_all(+Template, +Discriminator, :Goal, -Result)

Aggregate bindings in Goal according to Template. The aggregate_all/4 version performs findall/3 followed by sort/2 on Goal. See aggregate_all/3 to understand why this predicate can fail.

foreach(:Generator, :Goal)

True when the conjunction of instances of Goal created from solutions for Generator is true. Except for term copying, this could be implemented as below.

foreach(Generator, Goal) :-
    findall(Goal, Generator, Goals),
    maplist(call, Goals).

The actual implementation uses findall/3 on a template created from the variables shared between Generator and Goal. Subsequently, it uses every instance of this template to instantiate Goal, call Goal and undo only the instantiation of the template and not other instantiations created by running Goal. Here is an example:

?- foreach(between(1,4,X), dif(X,Y)), Y = 5.
Y = 5.
?- foreach(between(1,4,X), dif(X,Y)), Y = 3.
false.

The predicate foreach/2 is mostly used if Goal performs backtrackable destructive assignment on terms. Attributed variables (underlying constraints) are an example. Another example of a backtrackable data structure is in library(hashtable). If we care only about the side effects (I/O, dynamic database, etc.) or the truth value of Goal, forall/2 is a faster and simpler alternative. If Goal instantiates its arguments it is will often fail as the argument cannot be instantiated to multiple values. It is possible to incrementally grow an argument:

?- foreach(between(1,4,X), member(X, L)).
L = [1,2,3,4|_].

Note that SWI-Prolog up to version 8.3.4 created copies of Goal using copy_term/2 for each iteration, this makes the current implementation unable to properly handle compound terms (in Goal’s arguments) that share variables with the Generator. As a workaround you can define a goal that does not use compound terms, like in this example:

mem(E,L) :-  % mem/2 hides the compound argument from foreach/2
   member(r(E),L).

?- foreach(  between(1,5,N), mem(N,L)).

[det]free_variables(:Generator, +Template, +VarList0, -VarList)

Find free variables in bagof/setof template. In order to handle variables properly, we have to find all the universally quantified variables in the Generator. All variables as yet unbound are universally quantified, unless

they occur in the template
they are bound by X^P, setof/3, or bagof/3

free_variables(Generator, Template, OldList, NewList) finds this set using OldList as an accumulator.

author: - Richard O'Keefe
- Jan Wielemaker (made some SWI-Prolog enhancements)
license: Public domain (from DEC10 library).
To be done: - Distinguish between control-structures and data terms.
- Exploit our built-in term_variables/2 at some places?

[semidet,multifile]sandbox:safe_meta(+Goal, -Called)

Declare the aggregate meta-calls safe. This cannot be proven due to the manipulations of the argument Goal.

LogicalCaptain said (2021-07-01T08:45:38):

0 vs failure

As expected, aggregate_all/3, which uses findall/3, yields 0 if there is nothing:

?- aggregate_all(sum(P),false,Payoff).
Payoff = 0.

But aggregate/3, which uses bagof/3, fails if there is nothing:

?- aggregate(sum(P),false,Payoff).
false.

Properly quantifying free variables in the goal of aggregate∕3

Do not forget to properly quantify free variables in the goal of aggregate/3 in the same way as you would do for bagof/3 and setof/3 (using the `X^` notation) or you will get unexpected results:

:- begin_tests(aggregate).

test("aggregate all, sum, member-of-list") :-
   aggregate_all(sum(P),member(P,[1,2,3]),Sum),
   assertion(Sum == 6).

test("aggregate all, sum, key-value of dict") :-
   aggregate_all(sum(Value),get_dict(_Key,_{a:1,b:2,c:3},Value),Sum),
   assertion(Sum == 6).
   
test("aggregate all, sum, but no values") :-
   aggregate_all(sum(P),member(P,[]),Sum),
   assertion(Sum == 0).

test("aggregate, sum, member-of-list") :-
   aggregate(sum(P),member(P,[1,2,3]),Sum),
   assertion(Sum == 6).

test("aggregate all, sum, key-value of dict, forgot to quantify, thus wrong result and open choicepoint",[nondet]) :-
   aggregate(sum(Value),get_dict(_Key,_Tag{a:1,b:2,c:3},Value),Sum),
   assertion(Sum == 1).
   
test("aggregate all, sum, key-value of dict, properly quantified") :-
   aggregate(sum(Value),Tag^Key^get_dict(Key,Tag{a:1,b:2,c:3},Value),Sum),
   assertion(Sum == 6).

test("aggregate all, sum, key-value of dict, call 'hiding' predicate") :-
   aggregate(sum(Value),hide(Value),Sum),
   assertion(Sum == 6).

hide(Value) :-
   get_dict(_Key,_Tag{a:1,b:2,c:3},Value).
   
test("aggregate, sum, but no values", fail) :-
   aggregate(sum(P),member(P,[]),_Sum).

:- end_tests(aggregate).

Usage example

An example using aggregates to find a worker-task assignment yielding maximum payoff (testing all possibilites):

                 % -------------------> tasks 1...MaxTask  % workers 1..maxWorker
payoff_matrix( [ [22, 9,47,24,37,25,25],                   % |
                 [45,49,38,13, 7,45,50],                   % |
                 [33,10,10,40,13,25,32],                   % |
                 [ 6, 5,48,36,36,15, 3] ] ).               % V

max_task(MaxTask) :-
   payoff_matrix(M),
   nth0(0,M,Row),
   length(Row,MaxTask).

max_worker(MaxWorker) :-
   payoff_matrix(M),
   length(M,MaxWorker).

% What is the Payoff of assigning worker Worker
% to task Task? This predicate can also be redone
% to generate all Worker/Task combinations though we
% don't use it that way.

payoff(Worker,Task,Payoff) :-
   payoff_matrix(M),
   nth1(Worker,M,Row),
   nth1(Task,Row,Payoff).

% Find the best assignment using aggregate/3

best(WitnessAssignment,MaxPayoff) :-
   aggregate(
      max(Payoff,Assignment),
      gen_assignment(Assignment,Payoff),
      max(MaxPayoff,WitnessAssignment)).

% Backtrackably generate an Assignment. It has the given Payoff

gen_assignment(Assignment,Payoff) :-
   max_task(MaxTask),
   max_worker(MaxWorker),
   bagof(X,between(1,MaxTask,X),Tasks),                  % Tasks now contains the task numbers 1...MaxTask
   bagof(X,between(1,MaxWorker,X),Workers),              % Workers now contains the worker numbers 1...MaxWorkers
   gen_assignment_2(Tasks,[],Workers,Assignment),
   aggregate_all(sum(P),member(_{worker:_,task:_,payoff:P},Assignment),Payoff). % Use aggregate_all to not care about free variables

% gen_assignment_2(Tasks,TasksPicked,WorkersLeft,Assignment)
%
% Generate an assignment (a list of dicts, with each dict a worker-task
% pairing) by successively (and backtrackably) picking a task for the next
% worker from the list of tasks that haven't been assigned yet,
% getting the payoff for that worker-task pairing and recursively moving
% on to the next worker.

gen_assignment_2(Tasks,TasksPicked,[W|Ws],[_{worker:W,task:T,payoff:P}|As]) :-
   member(T,Tasks),
   \+ member(T,TasksPicked),
   payoff(W,T,P),
   gen_assignment_2(Tasks,[T|TasksPicked],Ws,As).
gen_assignment_2(_,_,[],[]).