Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
151 commits
Select commit Hold shift + click to select a range
9726646
[WIP] Initial version of index insight cron job
shuangli-z Jul 22, 2025
26ab62c
merge
xinyual Jul 25, 2025
8196b0c
add index insight restful api
xinyual Jul 28, 2025
fd63f5c
fix next token
xinyual Jul 28, 2025
b4dfdf2
fix next token
xinyual Jul 28, 2025
bc24539
[WIP] initial version of IndexInsightTask design
shuangli-z Jul 31, 2025
917e2a4
add access controller
xinyual Jul 28, 2025
529565f
Parallelize LLM calls in FieldDescriptionTask by batching
shuangli-z Aug 1, 2025
fd661b6
add waiting logic when prerequisite tasks are in GENERATING status
shuangli-z Aug 4, 2025
32d95a3
remove MLIndexInsightType.ALL and Index Insight List in api since Ind…
shuangli-z Aug 6, 2025
e9469f9
Change logic of sample fetching
pengjunquan-l Aug 6, 2025
dfc7bc0
add put container
xinyual Aug 7, 2025
7ac914c
Implemented basic execute operation
shuangli-z Aug 7, 2025
904708e
Add log related index check task for index insight
pengjunquan-l Aug 6, 2025
663fbc3
update LogRelatedIndexCheckTask to match interface modifications
shuangli-z Aug 7, 2025
7e7021b
Add logic to execute prerequisite task(s) first
shuangli-z Aug 7, 2025
999d741
add put index isight container
xinyual Aug 7, 2025
b56e7ad
Remove redundant task status checks in GetIndexInsightTransportAction
shuangli-z Aug 7, 2025
6080857
add put/delete container API
xinyual Aug 7, 2025
61dd355
refactor: remove deprecated implementation of index insight cron job
shuangli-z Aug 6, 2025
885e76e
merge latest code
xinyual Aug 7, 2025
4e5aec9
pass tenant id and add agent execution logic
xinyual Aug 7, 2025
fb91919
merge from main
xinyual Aug 7, 2025
aece0ae
add aggregation
xinyual Aug 8, 2025
e26bbca
change dsl
xinyual Aug 8, 2025
12479ef
apply spotless
xinyual Aug 8, 2025
21bc51f
use agent for LLM calls and container for insights storage
shuangli-z Aug 8, 2025
0156b07
resolve statistical data processing bugs and other minor fixes
shuangli-z Aug 11, 2025
848bdea
refine statistical task
xinyual Aug 11, 2025
b93cb45
add license
shuangli-z Aug 11, 2025
51b14a2
spotless updates
shuangli-z Aug 11, 2025
a01efe6
remove uselessResponse
xinyual Aug 11, 2025
abd10a0
remove deprecated code
shuangli-z Aug 11, 2025
0cd0e6a
use correct return status
xinyual Aug 11, 2025
ed270ac
Merge pull request #7 from xinyual/removeUselessResponse
shuangli-z Aug 11, 2025
74c79cb
set order for StatisticalDataTask result
shuangli-z Aug 12, 2025
6cb2a9d
add UT
xinyual Aug 12, 2025
ccaa187
add failure cases
xinyual Aug 12, 2025
e72cbc4
add UT for get index insight action
xinyual Aug 12, 2025
1e68c6c
add UT for delete/put container
xinyual Aug 12, 2025
cd77be9
apply spotless
xinyual Aug 12, 2025
2329416
delete IndexDescriptionTask and modify parsing logic in FieldDescript…
shuangli-z Aug 12, 2025
5e3b49b
add UT for response and request
xinyual Aug 12, 2025
ff08e6d
add license
xinyual Aug 12, 2025
6a7cfa0
Merge pull request #8 from xinyual/addUTForIndexInsight
shuangli-z Aug 12, 2025
a60f594
address PR comments in FieldDescriptionTask.java
shuangli-z Aug 13, 2025
3eb7fa8
Merge branch 'opensearch-project:main' into addIndexInsightFeature
shuangli-z Aug 13, 2025
1a2f2a7
fix simple commit
xinyual Aug 14, 2025
075643d
optimize logic creating container
xinyual Aug 14, 2025
bb0dfa3
fix comment
xinyual Aug 14, 2025
ea2dd30
Merge pull request #9 from xinyual/fixComment
shuangli-z Aug 14, 2025
651c4b4
Reorganize methods in IndexInsightTask and address PR comments
shuangli-z Aug 14, 2025
f68c368
merge latest code
shuangli-z Aug 14, 2025
663ec34
apply spotless
shuangli-z Aug 14, 2025
7880eee
add IT
xinyual Aug 14, 2025
76f8ecf
apply spotless
xinyual Aug 14, 2025
96cb9d0
Merge pull request #10 from xinyual/addITForIndexInsight
shuangli-z Aug 14, 2025
96c8097
modify prompt in FieldDescriptionTask to avoid specific examples
shuangli-z Aug 14, 2025
bf15158
add merge helper for statistical data task
xinyual Aug 15, 2025
95738d5
add IT
xinyual Aug 15, 2025
1f7899b
Merge pull request #11 from xinyual/addMergeRules
shuangli-z Aug 15, 2025
64f8c57
use pre defined mapping
xinyual Aug 15, 2025
31519cc
apply spotless
xinyual Aug 15, 2025
d334ac3
Merge pull request #12 from xinyual/usePreviousInFieldDescription
shuangli-z Aug 15, 2025
ff1ca3f
add Tool
xinyual Aug 18, 2025
3900426
add index insight tool
xinyual Aug 18, 2025
78bb2c0
Merge pull request #13 from xinyual/addIndexInsightToolForLatestCode
shuangli-z Aug 18, 2025
0ab6daf
add UT for index insight and improve code structure
shuangli-z Aug 14, 2025
8f985ab
Merge branch 'opensearch-project:main' into addIndexInsightFeature
shuangli-z Aug 18, 2025
848ebf9
add more ut
xinyual Aug 19, 2025
81bf676
Merge pull request #14 from xinyual/addMoreUT
shuangli-z Aug 19, 2025
ed3706d
add LogRelatedIndexCheckTaskTests
pengjunquan-l Aug 19, 2025
89e95de
revert useless change
xinyual Aug 19, 2025
9c4aefa
Merge pull request #15 from xinyual/revertuselessChange
shuangli-z Aug 19, 2025
8fec9ae
add UT for IndexInsightTask interface and its implementations
shuangli-z Aug 18, 2025
5e52f24
Merge pull request #16 from pengjunquan-l/addIndexInsightFeature
shuangli-z Aug 20, 2025
15f307e
Merge branch 'opensearch-project:main' into addIndexInsightFeature
shuangli-z Aug 20, 2025
8a12698
resolve comments
xinyual Aug 20, 2025
c2b6037
Merge pull request #17 from xinyual/resolveComment
shuangli-z Aug 20, 2025
ead5c50
fix UT
xinyual Aug 20, 2025
4d58d74
apply spotless
xinyual Aug 20, 2025
ee07454
Merge pull request #18 from xinyual/resolveComment
shuangli-z Aug 20, 2025
fabe2cf
address CR comments
shuangli-z Aug 20, 2025
484a11e
address CR comments
shuangli-z Aug 20, 2025
004d2e6
fix UT
shuangli-z Aug 20, 2025
78ccf78
address PR comments
shuangli-z Aug 21, 2025
e538426
change container name keyword
xinyual Aug 21, 2025
aba8f0a
add try catch exception
xinyual Aug 21, 2025
bbbf382
Merge pull request #20 from xinyual/fixCommentForXun
shuangli-z Aug 21, 2025
59aca4f
add more tests for LogRelatedIndexCheckTaskTests
pengjunquan-l Aug 21, 2025
4685c3b
Merge pull request #21 from pengjunquan-l/addIndexInsightFeature
shuangli-z Aug 21, 2025
050c7ce
add UT for LogRelatedIndexCheckTaskTests and refine other tests
shuangli-z Aug 21, 2025
da8f719
modify put api to put index insight config
xinyual Aug 21, 2025
882b28c
modify put api to put index insight config
xinyual Aug 21, 2025
032eeb6
add IndexInsightTestHelper to reduce code duplication
shuangli-z Aug 21, 2025
1b37b10
modifying API
xinyual Aug 22, 2025
90d4373
add get config API
xinyual Aug 22, 2025
e55811b
rename
xinyual Aug 22, 2025
101dcb4
add MLIndexInsightType.ALL
shuangli-z Aug 22, 2025
cdbc01e
add license
xinyual Aug 22, 2025
bb792f6
fix UT
xinyual Aug 22, 2025
8d306d3
add UT
xinyual Aug 22, 2025
b16e76e
fix UT
xinyual Aug 22, 2025
f585d59
add UT for MLIndexInsightType.ALL
shuangli-z Aug 22, 2025
845f208
fix IT
xinyual Aug 22, 2025
f9f83c5
add UT
xinyual Aug 25, 2025
39d0f2f
add IT
xinyual Aug 25, 2025
8318259
merge
xinyual Aug 25, 2025
4190380
rename
xinyual Aug 25, 2025
44224ed
Merge pull request #22 from xinyual/modifyAPI
shuangli-z Aug 25, 2025
5d3f6d1
refine code structure for MLIndexInsightType.ALL execution
shuangli-z Aug 25, 2025
602c9cd
filter useless in statistical and field description
xinyual Aug 27, 2025
085c569
fix statistical threshold
xinyual Aug 27, 2025
f6546f7
fix UT
xinyual Aug 27, 2025
625abde
fix IT
xinyual Aug 27, 2025
36ac7de
apply spotless
xinyual Aug 28, 2025
d3de92b
set to very low result
xinyual Aug 28, 2025
b0da256
Merge pull request #23 from xinyual/filterUselessColumn
shuangli-z Aug 28, 2025
ab752b8
remove useless IT
xinyual Aug 29, 2025
a115d4d
Merge pull request #24 from xinyual/removeUselessIT
shuangli-z Aug 29, 2025
ea713fd
add script build
xinyual Sep 2, 2025
b148f96
add pattern match
xinyual Sep 2, 2025
0daea3b
Merge pull request #25 from xinyual/addPatternMatch
shuangli-z Sep 3, 2025
0dcced3
move storage to system index
xinyual Sep 3, 2025
0dae01a
refine pattern match logic
shuangli-z Sep 3, 2025
049fe89
use sdkclient to search/get/put index insight
xinyual Sep 3, 2025
0d2dda2
fix UT
shuangli-z Sep 3, 2025
84b15a0
merge from latest
xinyual Sep 3, 2025
7719d90
address PR comments
shuangli-z Sep 4, 2025
35bbe59
fix UT
xinyual Sep 4, 2025
e511002
fix UT
xinyual Sep 4, 2025
27c6c78
merge from latest code
xinyual Sep 4, 2025
9a772c7
fix UT
xinyual Sep 4, 2025
abd67aa
apply spotless
xinyual Sep 4, 2025
20bbd4b
Merge pull request #26 from xinyual/changeToSystemIndex
shuangli-z Sep 4, 2025
d55acfc
fix IT
xinyual Sep 5, 2025
55d998e
Merge pull request #27 from xinyual/fixIT
shuangli-z Sep 5, 2025
3c1ce7e
fixLogic
xinyual Sep 5, 2025
e76c82c
Merge pull request #28 from xinyual/fixLogic
shuangli-z Sep 5, 2025
6cfb2ff
add and fix UT
shuangli-z Sep 4, 2025
928bcdd
fix comment for zna
xinyual Sep 5, 2025
c2bd0b0
remove useless cide
xinyual Sep 5, 2025
868be33
fix comment 2
xinyual Sep 5, 2025
4ec273f
fix error message logic
xinyual Sep 5, 2025
6832d69
Merge pull request #29 from xinyual/fixCommentZan
shuangli-z Sep 5, 2025
e21c35a
Merge branch 'opensearch-project:main' into addIndexInsightFeature
shuangli-z Sep 5, 2025
87cc151
add one more ut
xinyual Sep 5, 2025
e60f5e3
Merge pull request #30 from xinyual/addSTUT
shuangli-z Sep 5, 2025
359d55c
add ut for tool
xinyual Sep 5, 2025
1629148
Merge pull request #31 from xinyual/addToolUT
shuangli-z Sep 5, 2025
9d89155
remove deprecated variable storageIndex
shuangli-z Sep 5, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions common/src/main/java/org/opensearch/ml/common/CommonValue.java
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ public class CommonValue {
public static String HOT_BOX_TYPE = "hot";
// warm node
public static String WARM_BOX_TYPE = "warm";
public static final String ML_INDEX_INSIGHT_CONFIG_INDEX = ".plugins-ml-index-insight-config";
public static final String ML_INDEX_INSIGHT_STORAGE_INDEX = ".plugins-ml-index-insight-storage";

public static final String ML_MODEL_GROUP_INDEX = ".plugins-ml-model-group";
public static final String ML_MODEL_INDEX = ".plugins-ml-model";
public static final String ML_TASK_INDEX = ".plugins-ml-task";
Expand Down Expand Up @@ -68,6 +71,8 @@ public class CommonValue {
public static final String ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH = "index-mappings/ml_mcp_session_management.json";
public static final String ML_MCP_TOOLS_INDEX_MAPPING_PATH = "index-mappings/ml_mcp_tools.json";
public static final String ML_JOBS_INDEX_MAPPING_PATH = "index-mappings/ml_jobs.json";
public static final String ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH = "index-mappings/ml_index_insight_config.json";
public static final String ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH = "index-mappings/ml_index_insight_storage.json";

// Calculate Versions independently of OpenSearch core version
public static final Version VERSION_2_11_0 = Version.fromString("2.11.0");
Expand Down Expand Up @@ -111,4 +116,9 @@ public class CommonValue {

// TOOL Constants
public static final String TOOL_INPUT_SCHEMA_FIELD = "input_schema";

public static final String INDEX_INSIGHT_AGENT_NAME = "os_index_insight_agent";
public static final long INDEX_INSIGHT_GENERATING_TIMEOUT = 3 * 60 * 1000;
public static final long INDEX_INSIGHT_UPDATE_INTERVAL = 24 * 60 * 60 * 1000;

}
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,10 @@
import static org.opensearch.ml.common.CommonValue.ML_CONNECTOR_INDEX_MAPPING_PATH;
import static org.opensearch.ml.common.CommonValue.ML_CONTROLLER_INDEX;
import static org.opensearch.ml.common.CommonValue.ML_CONTROLLER_INDEX_MAPPING_PATH;
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_CONFIG_INDEX;
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH;
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_STORAGE_INDEX;
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH;
import static org.opensearch.ml.common.CommonValue.ML_JOBS_INDEX;
import static org.opensearch.ml.common.CommonValue.ML_JOBS_INDEX_MAPPING_PATH;
import static org.opensearch.ml.common.CommonValue.ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH;
Expand Down Expand Up @@ -50,7 +54,9 @@ public enum MLIndex {
MEMORY_CONTAINER(ML_MEMORY_CONTAINER_INDEX, false, ML_MEMORY_CONTAINER_INDEX_MAPPING_PATH),
MCP_SESSION_MANAGEMENT(MCP_SESSION_MANAGEMENT_INDEX, false, ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH),
MCP_TOOLS(MCP_TOOLS_INDEX, false, ML_MCP_TOOLS_INDEX_MAPPING_PATH),
JOBS(ML_JOBS_INDEX, false, ML_JOBS_INDEX_MAPPING_PATH);
JOBS(ML_JOBS_INDEX, false, ML_JOBS_INDEX_MAPPING_PATH),
INDEX_INSIGHT_CONFIG(ML_INDEX_INSIGHT_CONFIG_INDEX, false, ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH),
INDEX_INSIGHT_STORAGE(ML_INDEX_INSIGHT_STORAGE_INDEX, false, ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH);

private final String indexName;
// whether we use an alias for the index
Expand Down
Loading