TINKERPOP-3196 Split bulked traversers for LocalStep

CHANGELOG.asciidoc

@@ -35,6 +35,7 @@ This release also includes changes from <<release-3-7-XXX, 3.7.XXX>>.
 * Renamed `none()` step to `discard()`.
 * Repurposed `none()` step as a list filtering step with the signature `none(P)`.
 * Modified mathematical operators to prevent overflows in steps such as `sum()` and 'sack()' to prefer promotion to the next highest number type.
+* Modified `local()` to be "object-local" rather than "traverser-local".
 * Added `DateTime` ontop of the existing 'datetime' grammar.
 * Added `UUID()` and `UUID(value)` to grammar.
 * Deprecated the `UnifiedChannelizer`.

docs/src/dev/provider/gremlin-semantics.asciidoc

@@ -1619,6 +1619,36 @@ See: link:https://github.com/apache/tinkerpop/tree/x.y.z/gremlin-core/src/main/j
 link:https://github.com/apache/tinkerpop/tree/x.y.z/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/LengthLocalStep.java[source (local)],
 link:https://tinkerpop.apache.org/docs/x.y.z/reference/#length-step[reference]
 
+[[local-step]]
+=== local()
+
+*Description:* Executes the provided traversal in an object-local manner.
+
+*Syntax:* `local(Traversal localTraversal)`
+
+[width="100%",options="header"]
+|=========================================================
+|Start Step |Mid Step |Modulated |Domain |Range
+|N |Y |N |`any` |`any`
+|=========================================================
+
+*Arguments:*
+
+* `localTraversal` - The traversal that processes each single-object traverser individually.
+
+*Modulation:*
+
+None
+
+*Considerations:*
+
+The `local()` step enforces object-local execution. As a branching step with local children, it implements strict lazy
+evaluation by passing a single traverser at a time to the local traversal (bulk of exactly one, if bulking is supported)
+and resetting the traversal to clean state between executions.
+
+See: link:https://github.com/apache/tinkerpop/tree/x.y.z/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/LocalStep.java[source],
+link:https://tinkerpop.apache.org/docs/x.y.z/reference/#local-step[reference]
+
 [[intersect-step]]
 === intersect()
 

docs/src/reference/the-traversal.asciidoc

@@ -2553,30 +2553,15 @@ in an object-local traversal. As such, the `order().by()` and the `limit()` refe
 stream as a whole.
 
 Local Step is quite similar in functionality to <<general-steps,Flat Map Step>> where it can often be confused.
-`local()` propagates the traverser through the internal traversal as is without splitting/cloning it. Thus, its
-a “global traversal” with local processing. Its use is subtle and primarily finds application in compilation
-optimizations (i.e. when writing `TraversalStrategy` implementations. As another example consider:
+The primary distinction between these steps is that while `local()` preserves the path history of traversers as they
+pass through its child traversal, `flatMap()` does not. As another example consider:
 
 [gremlin-groovy,modern]
 ----
-g.V().both().barrier().flatMap(groupCount().by("name"))
-g.V().both().barrier().local(groupCount().by("name"))
+g.V().local(outE().inV()).path()
+g.V().flatMap(outE().inV()).path()
 ----
 
-Use of `local()` is often a mistake. This is especially true when its argument contains a reducing step. For example,
-let's say the requirement was to count the number of properties per `Vertex` in:
-
-[gremlin-groovy,modern]
-----
-g.V().both().local(properties('name','age').count())  <1>
-g.V().both().map(properties('name','age').count()) <2>
-----
-
-<1> The output here seems impossible because no single vertex in the "modern" graph can have more than two properties
-given the "name" and "age" filters, but because the counting is happening object-local the counting is occurring unique
-to each object rather than each global traverser.
-<2> Replacing `local()` with `map()` returns the result desired by the requirement.
-
 WARNING: The anonymous traversal of `local()` processes the current object "locally." In OLAP, where the atomic unit
 of computing is the vertex and its local "star graph," it is important that the anonymous traversal does not leave
 the confines of the vertex's star graph. In other words, it can not traverse to an adjacent vertex's properties or edges.

docs/src/upgrade/release-3.8.x.asciidoc

@@ -457,6 +457,34 @@ compatibility.
 
 See: link:https://issues.apache.org/jira/browse/TINKERPOP-3161[TINKERPOP-3161]
 
+==== Split bulked traversers for `local()`
+
+Prior to 3.8.0, local() exhibited "traverser-local" semantics, where the local traversal would apply independently to
+each individual bulkable `Traverser`. This often led to confusion, especially in the presence of reducing barrier steps, as
+bulked traversers would cause multiple objects to be processed at once. local() has been updated to automatically split
+any bulked traversers and thus now exhibits true "object-local" semantics.
+
+[source,groovy]
+----
+// 3.7.4
+gremlin> g.V().out().barrier().local(count())
+==>3
+==>1
+==>1
+==>1
+
+// 3.8.0
+gremlin> g.V().out().barrier().local(count())
+==>1
+==>1
+==>1
+==>1
+==>1
+==>1
+----
+
+See: link:https://issues.apache.org/jira/browse/TINKERPOP-3196[TINKERPOP-3196]
+
 ==== Removal of P.getOriginalValue()
 
 `P.getOriginalValue()` has been removed as it was not offering much value and was often confused with `P.getValue()`.

gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/branch/LocalStep.java

@@ -23,6 +23,7 @@
 import org.apache.tinkerpop.gremlin.process.traversal.step.TraversalParent;
 import org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep;
 import org.apache.tinkerpop.gremlin.process.traversal.traverser.TraverserRequirement;
+import org.apache.tinkerpop.gremlin.process.traversal.traverser.util.EmptyTraverser;
 import org.apache.tinkerpop.gremlin.process.traversal.util.FastNoSuchElementException;
 import org.apache.tinkerpop.gremlin.structure.util.StringFactory;
 
@@ -38,6 +39,7 @@ public final class LocalStep<S, E> extends AbstractStep<S, E> implements Travers
 
     private Traversal.Admin<S, E> localTraversal;
     private boolean first = true;
+    private Traverser.Admin<S> currentStart = EmptyTraverser.instance();
 
     public LocalStep(final Traversal.Admin traversal, final Traversal.Admin<S, E> localTraversal) {
         super(traversal);
@@ -58,25 +60,40 @@ public Set<TraverserRequirement> getRequirements() {
     protected Traverser.Admin<E> processNextStart() throws NoSuchElementException {
         if (this.first) {
             this.first = false;
-            this.localTraversal.addStart(this.starts.next());
+            this.localTraversal.addStart(nextStart());
         }
         while (true) {
             if (this.localTraversal.hasNext())
                 return this.localTraversal.nextTraverser();
-            else if (this.starts.hasNext()) {
+            else if (hasStartRemaining()) {
                 this.localTraversal.reset();
-                this.localTraversal.addStart(this.starts.next());
+                this.localTraversal.addStart(nextStart());
             } else {
                 throw FastNoSuchElementException.instance();
             }
         }
     }
 
+    private boolean hasStartRemaining() {
+        return (currentStart.bulk() > 0L) || this.starts.hasNext();
+    }
+
+    private Traverser.Admin<S> nextStart() throws NoSuchElementException {
+        if (currentStart.bulk() == 0L) {
+            currentStart = starts.next();
+        }
+        final Traverser.Admin<S> split = currentStart.split();
+        split.setBulk(1L);
+        currentStart.setBulk(currentStart.bulk() - 1L);
+        return split;
+    }
+
     @Override
     public void reset() {
         super.reset();
         this.first = true;
         this.localTraversal.reset();
+        this.currentStart = EmptyTraverser.instance();
     }
 
     @Override

gremlin-core/src/test/java/org/apache/tinkerpop/gremlin/process/traversal/step/branch/LocalStepTest.java

@@ -21,12 +21,16 @@
 import org.apache.tinkerpop.gremlin.process.traversal.Traversal;
 import org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__;
 import org.apache.tinkerpop.gremlin.process.traversal.step.StepTest;
+import org.junit.Test;
 
 import java.util.Arrays;
+import java.util.HashMap;
 import java.util.List;
+import java.util.Map;
 
 import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.in;
 import static org.apache.tinkerpop.gremlin.process.traversal.dsl.graph.__.out;
+import static org.junit.Assert.assertEquals;
 
 /**
  * @author Daniel Kuppitz (http://gremlin.guru)
@@ -41,4 +45,22 @@ protected List<Traversal> getTraversals() {
                 __.local(in())
         );
     }
+
+    @Test
+    public void shouldBeDebulkedToGroupCountSideEffectInLocal() {
+        final Traversal t = __.inject(1L, 1L, 2L, 3L).barrier().local(__.groupCount("x").select("x"));
+        assertEquals(Map.of(1L, 1L), t.next());
+        assertEquals(Map.of(1L, 2L), t.next());
+        assertEquals(Map.of(1L, 2L, 2L, 1L), t.next());
+        assertEquals(Map.of(1L, 2L, 2L, 1L, 3L, 1L), t.next());
+    }
+
+    @Test
+    public void shouldBeDebulkedToGroupCountInLocal() {
+        final Traversal t = __.inject(1L, 1L, 2L, 3L).barrier().local(__.groupCount());
+        assertEquals(Map.of(1L, 1L), t.next());
+        assertEquals(Map.of(1L, 1L), t.next());
+        assertEquals(Map.of(2L, 1L), t.next());
+        assertEquals(Map.of(3L, 1L), t.next());
+    }
 }

gremlin-dotnet/test/Gremlin.Net.IntegrationTest/Gherkin/Gremlin.cs

@@ -117,6 +117,11 @@ private static IDictionary<string, List<Func<GraphTraversalSource, IDictionary<s
                {"g_VX4X_localXbothE_limitX2XX_otherV_name", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V(p["vid4"]).Local<object>(__.BothE().Limit<object>(2)).OtherV().Values<object>("name")}}, 
                {"g_V_localXinEXknowsX_limitX2XX_outV_name", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().Local<object>(__.InE("knows").Limit<object>(2)).OutV().Values<object>("name")}}, 
                {"g_V_localXmatchXproject__created_person__person_name_nameX_selectXname_projectX_by_byXnameX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().Local<object>(__.Match<object>(__.As("project").In("created").As("person"), __.As("person").Values<object>("name").As("name"))).Select<object>("name", "project").By().By("name")}}, 
+               {"g_V_in_barrier_localXcountX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().In().Barrier().Local<object>(__.Count())}}, 
+               {"g_V_localXout_in_simplePathX_path", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().Local<object>(__.Out().In().SimplePath()).Path()}}, 
+               {"g_withSackX0LX_V_in_barrier_localXsackXsumX_byXageXX_sack", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.WithSack(0l).V().In().Barrier().Local<object>(__.Sack(Operator.Sum).By("age")).Sack<object>()}}, 
+               {"g_V_localXout_localXcountXX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().Local<object>(__.Out().Local<object>(__.Count()))}}, 
+               {"g_V_unionXoutE_count_localXinE_countXX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().Union<object>(__.OutE().Count(), __.Local<object>(__.InE().Count()))}}, 
                {"g_VX2X_optionalXoutXknowsXX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V(p["vid2"]).Optional<object>(__.Out("knows"))}}, 
                {"g_VX2X_optionalXinXknowsXX", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V(p["vid2"]).Optional<object>(__.In("knows"))}}, 
                {"g_V_hasLabelXpersonX_optionalXoutXknowsX_optionalXoutXcreatedXXX_path", new List<Func<GraphTraversalSource, IDictionary<string, object>, ITraversal>> {(g,p) =>g.V().HasLabel("person").Optional<object>(__.Out("knows").Optional<object>(__.Out("created"))).Path()}}, 

gremlin-go/driver/cucumber/gremlin.go

@@ -87,6 +87,11 @@ var translationMap = map[string][]func(g *gremlingo.GraphTraversalSource, p map[
     "g_VX4X_localXbothE_limitX2XX_otherV_name": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V(p["vid4"]).Local(gremlingo.T__.BothE().Limit(2)).OtherV().Values("name")}}, 
     "g_V_localXinEXknowsX_limitX2XX_outV_name": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().Local(gremlingo.T__.InE("knows").Limit(2)).OutV().Values("name")}}, 
     "g_V_localXmatchXproject__created_person__person_name_nameX_selectXname_projectX_by_byXnameX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().Local(gremlingo.T__.Match(gremlingo.T__.As("project").In("created").As("person"), gremlingo.T__.As("person").Values("name").As("name"))).Select("name", "project").By().By("name")}}, 
+    "g_V_in_barrier_localXcountX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().In().Barrier().Local(gremlingo.T__.Count())}}, 
+    "g_V_localXout_in_simplePathX_path": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().Local(gremlingo.T__.Out().In().SimplePath()).Path()}}, 
+    "g_withSackX0LX_V_in_barrier_localXsackXsumX_byXageXX_sack": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.WithSack(int64(0)).V().In().Barrier().Local(gremlingo.T__.Sack(gremlingo.Operator.Sum).By("age")).Sack()}}, 
+    "g_V_localXout_localXcountXX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().Local(gremlingo.T__.Out().Local(gremlingo.T__.Count()))}}, 
+    "g_V_unionXoutE_count_localXinE_countXX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().Union(gremlingo.T__.OutE().Count(), gremlingo.T__.Local(gremlingo.T__.InE().Count()))}}, 
     "g_VX2X_optionalXoutXknowsXX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V(p["vid2"]).Optional(gremlingo.T__.Out("knows"))}}, 
     "g_VX2X_optionalXinXknowsXX": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V(p["vid2"]).Optional(gremlingo.T__.In("knows"))}}, 
     "g_V_hasLabelXpersonX_optionalXoutXknowsX_optionalXoutXcreatedXXX_path": {func(g *gremlingo.GraphTraversalSource, p map[string]interface{}) *gremlingo.GraphTraversal {return g.V().HasLabel("person").Optional(gremlingo.T__.Out("knows").Optional(gremlingo.T__.Out("created"))).Path()}}, 

gremlin-python/src/main/python/radish/gremlin.py

@@ -90,6 +90,11 @@
     'g_VX4X_localXbothE_limitX2XX_otherV_name': [(lambda g, vid4=None:g.V(vid4).local(__.both_e().limit(2)).other_v().values('name'))], 
     'g_V_localXinEXknowsX_limitX2XX_outV_name': [(lambda g:g.V().local(__.in_e('knows').limit(2)).out_v().values('name'))], 
     'g_V_localXmatchXproject__created_person__person_name_nameX_selectXname_projectX_by_byXnameX': [(lambda g:g.V().local(__.match(__.as_('project').in_('created').as_('person'), __.as_('person').values('name').as_('name'))).select('name', 'project').by().by('name'))], 
+    'g_V_in_barrier_localXcountX': [(lambda g:g.V().in_().barrier().local(__.count()))], 
+    'g_V_localXout_in_simplePathX_path': [(lambda g:g.V().local(__.out().in_().simple_path()).path())], 
+    'g_withSackX0LX_V_in_barrier_localXsackXsumX_byXageXX_sack': [(lambda g:g.with_sack(long(0)).V().in_().barrier().local(__.sack(Operator.sum_).by('age')).sack())], 
+    'g_V_localXout_localXcountXX': [(lambda g:g.V().local(__.out().local(__.count())))], 
+    'g_V_unionXoutE_count_localXinE_countXX': [(lambda g:g.V().union(__.out_e().count(), __.local(__.in_e().count())))], 
     'g_VX2X_optionalXoutXknowsXX': [(lambda g, vid2=None:g.V(vid2).optional(__.out('knows')))], 
     'g_VX2X_optionalXinXknowsXX': [(lambda g, vid2=None:g.V(vid2).optional(__.in_('knows')))], 
     'g_V_hasLabelXpersonX_optionalXoutXknowsX_optionalXoutXcreatedXXX_path': [(lambda g:g.V().has_label('person').optional(__.out('knows').optional(__.out('created'))).path())], 

gremlin-test/src/main/resources/org/apache/tinkerpop/gremlin/test/features/branch/Choose.feature

@@ -337,10 +337,14 @@ Feature: Step - choose()
     When iterated to list
     Then the result should be unordered
       | result |
-      | l[marko,marko] |
-      | l[vadas,vadas] |
-      | l[josh,josh] |
-      | l[peter,peter] |
+      | l[marko] |
+      | l[marko] |
+      | l[vadas] |
+      | l[vadas] |
+      | l[josh] |
+      | l[josh] |
+      | l[peter] |
+      | l[peter] |
 
   @GraphComputerVerificationMidVNotSupported
   Scenario: g_unionXV_VXhasLabelXpersonX_barrier_mapXchooseXageX_optionXbetweenX26_30X_name_foldX_optionXnone_name_foldXX