Bind to source items by indicies

[atifaziz]

I'd like to suggest adding an operator that will return items of a sequence at given sequence of zero-based positions or indices. It will traverse the source items and an indices sequence in order and return the items at the corresponding indices. Duplicate indices are allowed. For most optimal (unbuffered) operation, indices would be given in increasing order.

The signature would be as follows:

public static IEnumerable<TResult>
    BindByIndex<T, TResult>(this IEnumerable<T> source, IEnumerable<int> indices,
        int lookBackSize,
        Func<int, TResult> missingSelector,
        Func<T, int, TResult> resultSelector)

Simpler overloads could be added.

If indices are in ascending order then lookBackSize is immaterial. If indices are unordered then lookBackSize will be the size of an internal scrolling buffer of source items that enables looking back.

The missingSelector argument allows projection of a result item when an index is not found, such as when it is less than zero or greater or equal to the number of items in source.

Suppose the following:

var xs = new[] { "foo", "bar", "baz", "qux" };
var result = xs.BindByIndex(indices, lookBackSize, i => "?", (s, _) => s);

The table below shows what result will contain for given inputs of indices and lookBackSize:

#	indices	lookBackSize	result
1	[2, 3]	0	[baz, qux]
2	[1, 3]	0	[bar, qux]
3	[0, 2, 3]	0	[foo, baz, qux]
4	[0, 2, 3]	0	[foo, baz, qux]
5	[0, 1, 1]	0	[foo, bar, bar]
6	[3, 1, 2]	0	[qux, ?, ?]
7	[3, 1, 2]	4	[qux, bar, baz]
8	[3, 1, 2]	1	[qux, ?, baz]
9	[-1, 1, 2, 10]	0	[?, bar, baz, ?]

Things to note:

• In example 5, indexing and returning duplicate items is supported.
• In example 6, items for indices 1 and 2 are missed because look-back is not possible after 3 (lookBackSize is 0) but example 7 fixes that.
• In example 8, the item for index is missed since lookBackSize is 1 and so a look-back that far from index 3 is no longer possible.

Example

const string csv = @"
    # Generated using https://mockaroo.com/
    id,first_name,last_name,email,gender,ip_address
    1,Maggee,Hould,mhould0@ft.com,Female,158.221.234.250
    2,Judas,Vedekhov,jvedekhov1@google.co.uk,Male,26.25.8.252
    3,Sharity,Desquesnes,sdesquesnes2@accuweather.com,Female,27.224.140.230
    4,Della,Conant,dconant3@japanpost.jp,Female,229.74.161.94
    5,Sansone,Hardson,shardson4@weather.com,Male,51.154.224.38
    6,Lloyd,Cromley,lcromley5@wikipedia.org,Male,168.145.20.63
    7,Ty,Bamsey,tbamsey6@ca.gov,Male,129.204.46.174
    8,Hurlee,Dumphy,hdumphy7@skyrock.com,Male,95.17.55.115
    9,Andy,Vickarman,avickarman8@qq.com,Male,10.159.118.60
    10,Jerad,Kerley,jkerley9@miitbeian.gov.cn,Male,3.19.136.57
    ";
    
// Parse CSV into rows of fields with commented lines,
// those starting with pound or hash (#), removed.

var rows =
    from row in Regex.Split(csv.Trim(), "\r?\n")
    select row.Trim() into row
    where row.Length > 0 && row[0] != '#'
    select row.Trim().Split(',');

// Split header and data rows

var (header, data) =
    rows.Index()
        .Partition(e => e.Key == 0, 
                   (hr, dr) => (hr.Single().Value, from e in dr select e.Value));
                   
// Locate indices of headers

var bindings = Enumerable.ToArray(
    from h in new[] { "id", "email", "last_name", "first_name", "foo" }
    select Array.FindIndex(header, sh => sh == h));
    
// Bind to data using 

var objects =
    from row in data
    select row.BindByIndex(bindings, bindings.Length, i => null, (f, _) => f)
              .Fold((id, email, ln, fn, foo) => new
              {
                  Id        = int.Parse(id),
                  FirstName = fn,
                  LastName  = ln,
                  Email     = new MailAddress(email),
                  Foo       = foo,
              });

foreach (var obj in objects)
    Console.WriteLine(obj.ToString());

Output

{ Id = 1, FirstName = Maggee, LastName = Hould, Email = mhould0@ft.com, Foo =  }
{ Id = 2, FirstName = Judas, LastName = Vedekhov, Email = jvedekhov1@google.co.uk, Foo =  }
{ Id = 3, FirstName = Sharity, LastName = Desquesnes, Email = sdesquesnes2@accuweather.com, Foo =  }
{ Id = 4, FirstName = Della, LastName = Conant, Email = dconant3@japanpost.jp, Foo =  }
{ Id = 5, FirstName = Sansone, LastName = Hardson, Email = shardson4@weather.com, Foo =  }
{ Id = 6, FirstName = Lloyd, LastName = Cromley, Email = lcromley5@wikipedia.org, Foo =  }
{ Id = 7, FirstName = Ty, LastName = Bamsey, Email = tbamsey6@ca.gov, Foo =  }
{ Id = 8, FirstName = Hurlee, LastName = Dumphy, Email = hdumphy7@skyrock.com, Foo =  }
{ Id = 9, FirstName = Andy, LastName = Vickarman, Email = avickarman8@qq.com, Foo =  }
{ Id = 10, FirstName = Jerad, LastName = Kerley, Email = jkerley9@miitbeian.gov.cn, Foo =  }

[atifaziz]

A prototype of the implementation is shared in PR #552 because it felt too large to share in a comment here.

[fsateler]

For the ordered case, or when loopbackCount is large enough, this appears to be identical to:

var orderedIndices = indices.ToList(); // Or perhaps HashSet? It might be more efficient.
var result = source.Index().Where(pair => orderedIndices.Contains(pair.Key)).Select(pair => pair.Value);

Is this the same? If so, then this leaves only the case where loopbackCount is too small as the only new functionality. What would be a use case for this?

[atifaziz]

What would be a use case for this?

Was the comprehensive example (binding to CSV headers) I posted in the opening comment not sufficient?

Is this the same?

Yes, except if you look at the implementation in #552, it's rather non-trivial. Also, it doesn't require hash look-ups in the ordered case.

[fsateler]

What would be a use case for this?

Was the comprehensive example (binding to CSV headers) I posted in the opening comment not sufficient?

Well, I could use a Dictionary<string, int> to store the positions and then skip the new method:

// same code as in opening until the header and data are split

var headerPositions = (from key in new[] { "id", "email", "last_name", "first_name", "foo" }
                       let position = header.IndexOf(key) 
                       select new KeyValuePair<string, int>(key, position)
                      ).ToDictionary();

string GetValue(string[] row, int pos) {
    if (pos < 0) {
        return null;
    }
    return row[pos];
}

var objects =
    from row in data
    select new  {
      Id        = int.Parse(GetValue(row, headerPositions["id"])),
      FirstName = GetValue(row, headerPositions["first_name"]),
      LastName  = GetValue(row, headerPositions["last_name"]),
      Email     = new MailAddress(GetValue(row, headerPositions["email"])),
      Foo       = GetValue(row, headerPositions["foo"]),
    };  

I don't see the advantage of the new method over this approach.

Is this the same?

Yes, except if you look at the implementation in #552, it's rather non-trivial. Also, it doesn't require hash look-ups in the ordered case.

And why would requiring hash lookups be a problem?

[atifaziz]

So you want to repeat all those headers twice? Okay, I know you're going to say that we could put them in constant strings so it's not a huge deal, but still, you've got a lot of ceremony going on like GetValue that's embodied within this proposal.

And why would requiring hash lookups be a problem?

It wouldn't. The idea here was that the indices as well as the source are themselves query comprehensions and neither may have been materialised in advance. I guess I should have known better than to post a naïve example. 😉 That said, I exactly expected (and thus appreciate) this kind of challenging and feedback so please keep charging on!

[viceroypenguin]

Implemented and released in SuperLinq v4.1.0