askrene: handle maxparts better.

[rustyrussell]

This is the algorithm recommended by @Lagrang3 . Much is neatening and tweaking existing code, though.

1. Removal of unnecessary caching.
2. Simplify sorting code (sort array of flows directly, don't use index arrays)
3. make increase_flows() more general, so it can be used for > 2% increase.
4. Use increase_flows() to top up the largest-capacity flows if we hit maxparts.

[Lagrang3]

1. Removal of unnecessary caching.

Yes. Caching at this phase was overzealous, when in fact we spend most of the time in doing MCF/

2. Simplify sorting code (sort array of flows directly, don't use index arrays)

Indexing was a necessity after using cached channel information.

3. make increase_flows() more general, so it can be used for > 2% increase.

I like it. It is preferred to the previous approach in my opinion.

Looping and remove 1/n of the shortage at every iteration worries me a little bit because
the shortage amount decays exponentially but the factor decreases the more flows there are
and we won't stop until 10sats.
For example, it takes 7 iterations to reduce some shortage to 1% of its initial value if
there are 2 paths. 44 iterations for 10 paths and 459 iterations for 100 paths.
Also the complexity is the number of iterations times the number of paths, hence
2 * 7 = 14
10 * 44 = 440
100 * 459 = 45900
respectively.
In practice it will be ok since we don't expect too many paths
or the shortage to be too large.

4. Use increase_flows() to top up the largest-capacity flows if we hit maxparts.

I strongly believe that reduce_num_flows shoudl sort flows based on delivery amount
and keep the largest delivery amount first.

[rustyrussell]

Here's the diff for the reduce_flows:

diff --git a/plugins/askrene/refine.c b/plugins/askrene/refine.c
index 2ee3855e2f..118460eb9d 100644
--- a/plugins/askrene/refine.c
+++ b/plugins/askrene/refine.c
@@ -588,36 +598,37 @@ double flows_probability(const tal_t *ctx, struct route_query *rq,
 	return probability;
 }
 
-/* Compare flows by remaining capacity (note: they are not reserved, so this
- * capacity includes current capacity. */
-static int reverse_cmp_capacity(struct flow *const *fa, struct flow *const *fb,
-				struct route_query *rq)
-{
-	struct amount_msat amount_a, amount_b;
-
-	amount_a = flow_max_deliverable(rq, *fa);
-	amount_b = flow_max_deliverable(rq, *fb);
-	if (!amount_msat_deduct(&amount_a, (*fa)->delivers))
-		abort();
-	if (!amount_msat_deduct(&amount_b, (*fb)->delivers))
-		abort();
-
-	if (amount_msat_eq(amount_a, amount_b))
-		return 0;
-	if (amount_msat_greater(amount_a, amount_b))
-		return -1;
-	return 1;
-}
-
 const char *reduce_num_flows(const tal_t *ctx,
 			     const struct route_query *rq,
 			     struct flow ***flows,
 			     struct amount_msat deliver,
 			     size_t num_parts)
 {
-	/* Keep the largest-capacity flows */
+	/* Keep the largest flows (not as I originally implemented, the largest
+	 * capacity flows).  Here's Lagrang3's analysis:
+	 *
+	 * I think it is better to keep the largest-deliver flows.  If we only
+	 * go for the highest capacity we may throw away the low cost benefits
+	 * of the MCF.
+
+	 * Hypothetical scenario: MCF finds 3 flows but maxparts=2,
+	 * flow 1: deliver=10, cost=0, capacity=0
+	 * flow 2: deliver=7, cost=1, capacity=5
+	 * flow 3: deliver=1, cost=10, capacity=100
+	 *
+	 * It is better to keep flows 1 and 2 by accomodating 1 more unit of
+	 * flow in flow2 at 1 value expense (per flow), than to keep flows 2 and
+	 * 3 by accomodating 5 more units of flow in flow2 at cost 1 and 5 in
+	 * flow3 at cost 100.
+	 *
+	 * The trade-off is: if we prioritize the delivery value already
+	 * computed by MCF then we find better solutions, but we might fail to
+	 * find feasible solutions sometimes. If we prioritize capacity then we
+	 * generally find bad solutions though we find feasibility more often
+	 * than the alternative.
+	 */
 	size_t orig_num_flows = tal_count(*flows);
-	asort(*flows, orig_num_flows, reverse_cmp_capacity, cast_const(struct route_query *, rq));
+	asort(*flows, orig_num_flows, revcmp_flows, NULL);
 	tal_resize(flows, num_parts);
 
 	if (!increase_flows(rq, *flows, deliver, -1.0))