But why does list_to_set/2 demand 2 sorts?

Because, as the description says, we are checking for same elements *in the same order*.

You see by inspecting the source code:

list_to_set(List, Set) :-
    must_be(list, List),

    % preserve the original positions of elements
    number_list(List, 1, Numbered),

    % fisrt sort by value in order to remove duplicates
    sort(1, @=<, Numbered, ONum),
    remove_dup_keys(ONum, NumSet),

    % then sort by original position, since we're
    % comparing the order of the elements as well
    sort(2, @=<, NumSet, ONumSet),
    pairs_keys(ONumSet, Set).

See also library(ordset)

https://www.swi-prolog.org/pldoc/man?section=ordsets

and its predicate list_to_ord_set/2

https://www.swi-prolog.org/pldoc/doc_for?object=list_to_ord_set/2

which imposes sorting on the result set.

(If you look at the implementation of list_to_ord_set/2, it is just a call to sort/2)

?- list_to_set([3,1,2,3,1,4],L).
L = [3, 1, 2, 4].

?- list_to_ord_set([3,1,2,3,1,4],L).
L = [1, 2, 3, 4].  % internal structure is destroyed

% the above actually the same as...

?- sort([3,1,2,3,1,4],L).
L = [1, 2, 3, 4].  % internal structure is destroyed

(But why does list_to_set/2 demand 2 sorts? It is a mystery.)