Revamp docs as UBI has evolved.

_search-plugins/ubi/dsl-queries.md

@@ -95,7 +95,4 @@ GET ubi_events/_search
 ```
 {% include copy.html %}
 
-You can run the preceding queries in the OpenSearch Dashboards [Query Workbench]({{site.url}}{{site.baseurl}}/search-plugins/sql/workbench/). 
-
-A demo workbench with sample data can be found here: 
-[http://chorus-opensearch-edition.dev.o19s.com:5601/app/OpenSearch-query-workbench](http://chorus-OpenSearch-edition.dev.o19s.com:5601/app/OpenSearch-query-workbench).
+You can run the preceding queries in the OpenSearch Dashboards [Dev Tools]({{site.url}}{{site.baseurl}}/dev-tools/index-dev/) console.

_search-plugins/ubi/index.md

@@ -11,7 +11,7 @@
 **Introduced 2.15**
 {: .label .label-purple }
 
-**References UBI Specification 1.0.0**
+**References UBI Specification 1.3.0**
 {: .label .label-purple }
 
 User Behavior Insights (UBI) is a schema for capturing user search behavior. Search behavior consists of the queries that the user submits, the results that are presented to them, and the actions they take on those results. The UBI schema links all user interactions (events) to the search result they were performed on. That is, it not only captures the chronological sequence of events but also captures the causal links between events. Analysis of this behavior is used for improving the quality of search results.
@@ -20,33 +20,66 @@
 
 In principle, queries sent to the server and results returned by the server can be sent to the UBI endpoint from the client. But as an optimization, they can instead be sent directly to the UBI endpoint from the server, without incurring a round-trip to the client. That is the function of the UBI plugin and is not a requirement to adopt UBI.
 
-UBI includes the following elements:
-* A machine-readable [schema](https://github.com/o19s/ubi) that faciliates interoperablity of the UBI specification.
-* A client-side JavaScript [example reference implementation]({{site.url}}{{site.baseurl}}/search-plugins/ubi/data-structures/) that shows how to capture events and send them to the OpenSearch UBI plugin.
-* An OpenSearch [plugin](https://github.com/opensearch-project/user-behavior-insights) that captures server-side behavior.
-
-<!-- vale off -->
 
-The UBI documentation is organized into two categories: *Explanation and reference* and *Tutorials and how-to guides*:   
+> “how our users are using our product, whether search results were useful for them and whether they clicked on top-n results we gave and all related stuff” - Data Scientist working Search.
 
-*Explanation and reference*
-
-| Link | Description |
-| :--------- | :------- |
-| [UBI Request/Response Specification](https://github.com/o19s/ubi/) | The industry-standard schema for UBI requests and responses. The current version references UBI Specification 1.0.0.  |
-| [UBI index schema]({{site.url}}{{site.baseurl}}/search-plugins/ubi/schemas/) | Documentation on the individual OpenSearch query and event stores. |
+UBI includes the following elements:
+* A machine-readable [schema](https://github.com/o19s/ubi) that facilitates interoperability of the UBI specification.
+* [ubi.js](https://github.com/opensearch-project/user-behavior-insights/tree/main/ubi-javascript-collector/ubi.js): a (optional!) clientside JavaScript library for capturing searches and events.
+* An (optional!) OpenSearch [plugin](https://github.com/opensearch-project/user-behavior-insights) that streamlines the recording of query data.
 
+Advanced features in OpenSearch, such as the [Search Relevance Workbench]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/using-search-relevance-workbench/) and the [Hybrid Search Optimizer]({{site.url}}{{site.baseurl}}/search-plugins/search-relevance/optimize-hybrid-search/), build on the data collected according to the UBI specification.
 
-*Tutorials and how-to guides*
+<!-- vale off -->
 
-| Link | Description |
-| :--------- | :------- |
-| [UBI plugin](https://github.com/opensearch-project/user-behavior-insights) | How to install and use the UBI plugin. |
-| [UBI client data structures]({{site.url}}{{site.baseurl}}/search-plugins/ubi/data-structures/)  | Sample JavaScript structures for populating the event store. |
-| [Example UBI query DSL queries]({{site.url}}{{site.baseurl}}/search-plugins/ubi/dsl-queries/)  | How to write queries for UBI data in OpenSearch query DSL. |
-| [Example UBI SQL queries]({{site.url}}{{site.baseurl}}/search-plugins/ubi/sql-queries/)  | How to write analytic queries for UBI data in SQL. |
-| [UBI dashboard tutorial]({{site.url}}{{site.baseurl}}/search-plugins/ubi/ubi-dashboard-tutorial/) | How to build a dashboard containing UBI data. |
-| [Chorus Opensearch Edition](https://github.com/o19s/chorus-opensearch-edition/?tab=readme-ov-file#structured-learning-using-chorus-opensearch-edition) katas | A series of structured tutorials that teach you how to use UBI with OpenSearch through a demo e-commerce store. |
+<table>
+  <tr style="vertical-align: top;">
+    <td>
+      <h2>Tutorials</h2>
+      <ul>                                        
+        <li><a href="{{site.url}}{{site.baseurl}}/search-plugins/ubi/ubi-dashboard-tutorial/">Learn to create custom dashboards</a> for visualizing UBI data.</li>    
+        <li> Based on <a href="https://github.com/o19s/chorus-opensearch-edition">Chorus for OpenSearch</a> demo:
+          <ul>
+            <li><a href="https://github.com/o19s/chorus-opensearch-edition/blob/main/katas/002_derive_interaction_data.md">Derive Interaction Data from User Clicks.</a></li>
+            <li><a href="https://github.com/o19s/chorus-opensearch-edition/blob/main/katas/006_protecting_sensitive_information.md">Protecting sensistive information when using UBI.</a></li>
+            <li><a href="https://github.com/o19s/chorus-opensearch-edition/blob/main/katas/007_configure_AB_with_TDI.md">Configuring an AB test with Team Draft Interleaving</a></li>    
+          </ul>
+        </li>
+      </ul>
+    </td>
+    <td>
+      <h2>How To Guides</h2>
+      <ul>        
+        <li>How to <a href="https://github.com/opensearch-project/user-behavior-insights?tab=readme-ov-file#user-quick-start">install and use the UBI plugin</a> in OpenSearch.</li>          
+        <li>How to use <a href="{{site.url}}{{site.baseurl}}/search-plugins/ubi/ubi-javascript-collector/">ubi.js</a>, a client-side JavaScript library for capturing events.</li>
+        <li>How to <a href="{{site.url}}{{site.baseurl}}/search-plugins/ubi/dsl-queries/">write queries for UBI data using OpenSearch query DSL.</a></li>
+        <li>How to <a href="{{site.url}}{{site.baseurl}}/search-plugins/ubi/sql-queries/">write analytic queries for UBI data using SQL.</a></li>          
+      </ul>
+    </td>
+  </tr>
+  <tr style="vertical-align: top;">
+    <td>
+      <h2>Explanation</h2>
+      <ul>
+        <li><a href="https://docs.google.com/presentation/d/e/2PACX-1vTJ9wYhhRG2sHxB-pm2Pfcqv0AzwRzSgTn-VyKTV6bL4PyXQC9C9kE6Oyrkag2_Olb6Ugevs_kbflId/pub?start=true&loop=false&delayms=3000">Why UBI?</a> presentation.</li>
+        <li>Learn more about this standard via <a href="https://www.UBISearch.dev">https://www.UBISearch.dev</a>, the community clearinghouse.</li>                
+        <li>Watch <a href="https://youtu.be/0chun264PRQ">Leveraging UBI to enhance Search Relevance</a> talk to understand how to use this data to improve search quality.</li>
+        <li>Go deeper with UBI.  Watch <a href="https://www.youtube.com/watch?v=xi261oUamXc">You’ve Deployed User Behavior Insights. Now What?</a> to see what else you can do.</li>
+      </ul>
+    </td>
+    <td>
+        <h2>Reference</h2>
+        <ul>
+            <li><a href="https://github.com/opensearch-project/user-behavior-insights">UBI Plugin for OpenSearch</a></li>
+              <li><a href="{{site.url}}{{site.baseurl}}/search-plugins/ubi/schemas/">UBI Schema in OpenSearch</a></li>
+            <li>Repository for the <a href="https://github.com/o19s/ubi">UBI Schema</a>.</li>                
+            <li><a href="https://o19s.github.io/ubi/docs/html/1.3.0/query.request.schema.html">Query Tracking Specification</a></li>
+            <li><a href="https://o19s.github.io/ubi/docs/html/1.3.0/event.schema.html">Event Tracking Specification</a></li>                
+            
+        </ul>
+    </td>
+  </tr>
+</table>
 
 <!-- vale on -->
 The documentation categories were adapted using concepts based on [Diátaxis](https://diataxis.fr/).

_search-plugins/ubi/schemas.md

@@ -16,8 +16,8 @@ The User Behavior Insights (UBI) data collection process involves tracking and r
 
 For UBI to function properly, the connections between the following fields must be consistently maintained within an application that has UBI enabled:
 
-- [`object_id`](#object_id) represents an ID for whatever object the user receives in response to a query. For example, if you search for books, it might be an ISBN code of a book, such as `978-3-16-148410-0`.
-- [`query_id`](#query_id) is a unique ID for the raw query language executed and the `object_id` values of the _hits_ returned by the user's query.  
+- [`object_id`](#object_id) represents an ID for whatever object the user receives in response to a query. For example, if you search for books, it might be an ISBN for a book, such as `978-3-16-148410-0`.
+- [`query_id`](#query_id) is a unique ID for the raw query executed, while the `object_id` maps to the primary identifier of the _hits_ returned by the user's query. 
 - [`client_id`](#client_id) represents a unique query source. This is typically a web browser used by a unique user.
 - [`object_id_field`](#object_id_field) specifies the name of the field in your index that provides the `object_id`. For example, if you search for books, the value might be `isbn_code`.
 - [`action_name`](#action_name), though not technically an ID, specifies the exact user action (such as `click`, `add_to_cart`, `watch`, `view`, or `purchase`) that was taken (or not taken) for an object with a given `object_id`.
@@ -138,11 +138,11 @@ All underlying query information and results (`object_ids`) are stored in the `u
 
 The `ubi_queries` index [schema](https://github.com/OpenSearch-project/user-behavior-insights/tree/main/src/main/resources/queries-mapping.json) includes the following fields:
 
-- `timestamp` (events and queries): A UNIX timestamp indicating when the query was received.
+- `timestamp` (events and queries): An ISO 8601-formatted timestamp indicating when the query was received.
 
-- `query_id` (events and queries): The unique ID of the query provided by the client or generated automatically. Different queries with the same text generate different `query_id` values. 
-	
-- `client_id` (events and queries): A user/client ID provided by the client application.
+- `query_id` (events and queries): The unique ID of the query provided by the client or generated by the search engine. Different queries with the same text generate different `query_id` values.
+
+- `client_id` (events and queries): A client ID provided by the client application.
 
 - `query_response_objects_ids` (queries): An array of object IDs. An ID can have the same value as the `_id`, but it is meant to be the externally valid ID of a document, item, or product.
 
@@ -169,14 +169,14 @@ The following are the predefined, minimal fields in the `ubi_events` index:
  <p id="query_id"> </p>
 
 - `query_id` (size 100): The unique identifier of a query, which is typically a UUID but can be any string.
- The `query_id` is either provided by the client or generated at index time by the UBI plugin. The `query_id` values in both the **UBI queries** and **UBI events** indexes must be consistent.
+ The `query_id` is either provided by the client or generated at query time by the UBI plugin. The `query_id` values in both the **UBI queries** and **UBI events** indexes must be consistent.
 
 <p id="client_id"> </p>
 
 - `client_id`: The client that issues the query. This is typically a web browser used by a unique user.
  The `client_id` in both the **UBI queries** and **UBI events** indexes must be consistent.
 
-- `timestamp`: When the event occurred, either in UNIX format or formatted as `2018-11-13T20:20:39+00:00`.
+- `timestamp`: The time the event occurred in ISO 8601 format, such as `2018-11-13T20:20:39+00:00Z`.
 
 - `message_type` (size 100): A logical bin for grouping actions (each with an `action_name`). For example, `QUERY` or `CONVERSION`. 
 
@@ -193,18 +193,12 @@ The following are the predefined, minimal fields in the `ubi_events` index:
 
    - `event_attributes.position.ordinal`: Tracks the list position that a user can select (for example, selecting the third element can be described as `event{onClick, results[4]}`).
 
-    - `event_attributes.position.{x,y}`: Tracks x and y values defined by the client.
-
-    - `event_attributes.position.page_depth`: Tracks the page depth of the results.
-
-    - `event_attributes.position.scroll_depth`: Tracks the scroll depth of the page results.
-
-    - `event_attributes.position.trail`: A text field that tracks the path/trail that a user took to get to this location.
-
+    - `event_attributes.position.xy.{x,y}`: Tracks x and y values defined by the client.
+
   - `event_attributes.object`: Contains identifying information about the object returned by the query (for example, a book, product, or post).
    The `object` structure can refer to the object by internal ID or object ID. The `object_id` is the ID that links prior queries to this object. This field comprises the following subfields:
 
-    - `event_attributes.object.internal_id`: A unique ID that OpenSearch can use to internally index the object, for example, the `_id` field in the indexes.
+    - `event_attributes.object.internal_id`: The unique ID that OpenSearch uses to internally index the object, for example, the `_id` field in the indexes.
 
       <p id="object_id">
 
@@ -214,7 +208,7 @@ The following are the predefined, minimal fields in the `ubi_events` index:
 
       <p id="object_id_field">
 
-    - `event_attributes.object.object_id_field`: Indicates the type/class of the object and the name of the search index field that contains the `object_id`.  
+    - `event_attributes.object.object_id_field`: Indicates the type/class of the object and the name of the search index field that contains the `object_id` such as `ssn`, `isbn`, or `ean`.
 
     - `event_attributes.object.description`: An optional description of the object.