[Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. #131486

mashhurs · 2025-07-17T22:44:11Z

Why we are doing this?

A bit info about plugin
[elastic_integration](plugin https://github.com/elastic/logstash-filter-elastic_integration) retrieves ingest pipelines for the data stream and runs Elastic integrations it resolved. In order to do so, It internally embeds the ES sources (ingest processors, ingest plugins, etc...), will be regularly built against ES (with all active branches) and released as a [ruby gem]https://rubygems.org/gems/logstash-filter-elastic_integration

The problem we are facing
When ES interfaces change (for e.g, GeoIP), result in plugin build breakage. So, we in the end, came up with a solution to introduce an agreement between elastic_integration plugin and ES, logstash-bridge module! The module wraps up any classes/modules the plugin depends on and whenever interface changes, developers will apply directly to the bridge Wrapper interfaces will stay without change. This will guarantee the plugin stability.

What this PR does?

It migrates majority plugin interface usages to the bridge. This investigation shows the plugin usage details.
Exports following modules to make visible in logstash-bridge:

ingest-user-agent (org.elasticsearch.ingest.useragent)
mapper-constant-keyword (org.elasticsearch.xpack.constantkeyword)
x-pack:plugin:redact (org.elasticsearch.xpack.redact)
:x-pack:plugin:spatial (only required ones: org.elasticsearch.xpack.spatial, org.elasticsearch.xpack.spatial.common and org.elasticsearch.xpack.spatial.search.aggregations.bucket.geogrid)
:x-pack:plugin:wildcard (org.elasticsearch.xpack.wildcard)

… exception.

…h default method to avoid required interface implementation.

… the bridge. Processor definitions also moved to the bridge. Module simplofications.

…k spatial/wildcard/mapper-constant modules.

… module.

elasticsearchmachine · 2025-07-21T07:51:17Z

Pinging @elastic/es-core-infra (Team:Core/Infra)

- transitions terminology from wrap/unwrap to toInternal/fromInternal - adds abstract base class for ProcessorBridge, since we are expecting external implementations, which includes an internal-shaped proxy to the external definition. - adds copious commentary for the classes that were previously shipped

yaauie

I think this looks on the right track -- I opened mashhurs#1 to refactor things a bit, and have a comment about where the constants should live.

libs/logstash-bridge/src/main/java/org/elasticsearch/logstashbridge/ingest/ProcessorBridge.java

spike refactor of logstashbridge stable API

rjernst

This bridge is getting quite a bit more complex, now trying to mimic some initialization of plugins. I want to reiterate and confirm something from when we first added it: this bridge is entirely an internal artifact of Logstash, right? In no way should this artifact ever be published for end users.

x-pack/plugin/mapper-constant-keyword/src/main/java/module-info.java

x-pack/plugin/wildcard/src/main/java/module-info.java

…painless spi

mashhurs · 2025-07-22T21:46:11Z

This bridge is getting quite a bit more complex, now trying to mimic some initialization of plugins. I want to reiterate and confirm something from when we first added it: this bridge is entirely an internal artifact of Logstash, right? In no way should this artifact ever be published for end users.

👋 @rjernst, your understanding is correct that logstash-bridge is entirely internal, no end user exposure so far. From now end, https://github.com/elastic/logstash-filter-elastic_integration plugin aligns on this bridge w/o directly importing modules/classes from ES source.

This bridge is getting quite a bit more complex, now trying to mimic some initialization of plugins.

Historically, we got a customer failure (with below error) case which led us to initialize painless extensions (elastic/logstash-filter-elastic_integration#162) and this is where we need to make accessible (and open an access to its resources) [redact, wildcard, mapper-constant-keyword] modules.

logstash-1       | Caused by: java.lang.IllegalArgumentException: cannot resolve symbol [Processors]
logstash-1       |      at org.elasticsearch.painless.PainlessScript$Script.compile(long bytes = Processors.bytes(params['size']); ctx.size_in_bytes = bytes;:24) ~[?:?]
logstash-1       |      at org.elasticsearch.painless.phase.DefaultSemanticAnalysisPhase.visitCall(DefaultSemanticAnalysisPhase.java:3233)

Thank you for taking a look. We believe this modular descriptor change is beyond logstash-team scope. If you prefer to make this modular change with a separate PR (to get reviewed from corresponded team), I will make it happen but if your concern is about non-qualified opens statements, I have updated them to be specific (thank you for catching, I added it to spatial but somehow forgot adding to others).

rjernst

Thanks @mashhurs for modularizing these. The change looks fine, just one last ask on the module names.

x-pack/plugin/mapper-constant-keyword/src/main/java/module-info.java

…y that remove xpack from namings. Move constants from processor bridge to align with a proper place where Ingest common plugin bridge is sutable.

yaauie

Fantastic work. Thank you.

…king * upstream/main: (90 commits) Register a blob cache long counter metric for total evicted regions (elastic#131862) Move plan attribute resolution to its own component (elastic#131830) Make restore support multi-project (elastic#131661) Use logically more correct expression (elastic#131869) [ES|QL] Change equals and hashcode for ConstantNullBlock (elastic#131817) Update `TransportVersion` to support a new model (elastic#131488) Correct slow log user for RCS 2.0 (elastic#130140) Revert "Remove 8.17 from dev branches" Mute org.elasticsearch.compute.aggregation.ValuesBytesRefGroupingAggregatorFunctionTests testSomeFiltered elastic#131878 Remove 8.17 from dev branches Revert "CompressorFactory.compressor (elastic#131655)" (elastic#131866) Add fast path for single value in VALUES aggregator (elastic#130510) Resolve inference release tests failing due to missing feature flag (elastic#131841) [Docs] Replace placeholder URLs (elastic#131309) CompressorFactory.compressor (elastic#131655) add availability info for speed loading setting (elastic#131714) [Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. (elastic#131486) Migrate x-pack-enrich legacy rest tests to new test framework (elastic#131743) Fix plugin example test failures due to deprecation warning (elastic#131819) Remove deprecated function isNotNullAndFoldable (elastic#130944) ...

mashhurs added 9 commits July 14, 2025 09:32

Testing painless processors extension.

b9cdd5b

Test paintless spaial, wildcard and constant keyword extensions.

e3310cd

Fix the bug that when null applied for Map.of() which may cause NPE.

3a6a795

Debug

e2dd123

Debugging: require delegate and its metadata non null to get specific…

84f38e3

… exception.

Debugging: instead of Map.copyOf(), send ingest metadata as it is.

78a3394

Make metadata version field accessible, introduce inverse wrapper wit…

38b25ce

…h default method to avoid required interface implementation.

RedactPlugin, IngestCommonPlugin and IngestUserAgent plugins moved to…

0e70a6c

… the bridge. Processor definitions also moved to the bridge. Module simplofications.

Ingest common plugin simplification.

b7ac030

elasticsearchmachine added v9.2.0 external-contributor Pull request authored by a developer outside the Elasticsearch team labels Jul 17, 2025

elasticsearchmachine and others added 9 commits July 17, 2025 22:53

[CI] Auto commit changes from spotless

0d0acef

Open an access for the x-pack spatial module resources.

577e0f2

[CI] Auto commit changes from spotless

94aa3e2

Open access to resources in mapper constant keyword.

87774ae

Add an access to resource in x-pack wildcard module.

bfa2d27

Export and open access to sub-package spatial modules.

4b1ef11

Provide module service implementations with provides keyword in x-pac…

fb835ba

…k spatial/wildcard/mapper-constant modules.

Export x-pack spatial module packages to make accessible to ES server…

b4346d3

… module.

Merge branch 'main' into move-to-bridge-stable-api-investigation

a9981f5

mashhurs self-assigned this Jul 18, 2025

mashhurs marked this pull request as ready for review July 18, 2025 20:30

mashhurs requested a review from a team as a code owner July 18, 2025 20:30

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jul 18, 2025

mashhurs added the Team:Core/Infra Meta label for core/infra team label Jul 18, 2025

elasticsearchmachine removed the Team:Core/Infra Meta label for core/infra team label Jul 18, 2025

gbanasiak added the :Core/Infra/Plugins Plugin API and infrastructure label Jul 21, 2025

elasticsearchmachine added Team:Core/Infra Meta label for core/infra team and removed needs:triage Requires assignment of a team area label labels Jul 21, 2025

yaauie mentioned this pull request Jul 21, 2025

spike refactor of logstashbridge stable API mashhurs/elasticsearch#1

Merged

yaauie reviewed Jul 22, 2025

View reviewed changes

libs/logstash-bridge/src/main/java/org/elasticsearch/logstashbridge/ingest/ProcessorBridge.java Outdated Show resolved Hide resolved

Merge pull request #1 from yaauie/stable-api-clarity-external-internal

ba80d06

spike refactor of logstashbridge stable API

mashhurs mentioned this pull request Jul 22, 2025

[Move to ES bridge -> ES scope] Evaluate current status of Stable API based on plugin usage elastic/logstash-filter-elastic_integration#331

Closed

Format the recent logstash-bridge changes.

cd44d2f

mashhurs added the >feature label Jul 22, 2025

rjernst reviewed Jul 22, 2025

View reviewed changes

x-pack/plugin/mapper-constant-keyword/src/main/java/module-info.java Outdated Show resolved Hide resolved

x-pack/plugin/wildcard/src/main/java/module-info.java Outdated Show resolved Hide resolved

mashhurs and others added 2 commits July 22, 2025 14:15

Wildcard and mapper constant keyword modules open resource access to …

4eee343

…painless spi

Merge branch 'main' into move-to-bridge-stable-api-investigation

ede5de5

rjernst approved these changes Jul 22, 2025

View reviewed changes

x-pack/plugin/mapper-constant-keyword/src/main/java/module-info.java Outdated Show resolved Hide resolved

mashhurs and others added 3 commits July 22, 2025 23:02

Rename constant-keyword, wildcard, redact and spatial modules in a wa…

166d4be

…y that remove xpack from namings. Move constants from processor bridge to align with a proper place where Ingest common plugin bridge is sutable.

Merge branch 'main' into move-to-bridge-stable-api-investigation

c9b3be1

Remove unnecessary constant class in the ingest common plugin bridge.

7f84caf

mashhurs mentioned this pull request Jul 23, 2025

Move to ES bridge API elastic/logstash-filter-elastic_integration#336

Open

This was linked to issues Jul 23, 2025

[Move to ES bridge -> ES scope] Introduce module descriptors for the plugins elastic/logstash-filter-elastic_integration#352

Closed

[Move to ES bridge -> ES scope] Required changes after investigation elastic/logstash-filter-elastic_integration#353

Closed

yaauie approved these changes Jul 24, 2025

View reviewed changes

mashhurs merged commit 1e87f67 into elastic:main Jul 24, 2025
33 checks passed

mashhurs deleted the move-to-bridge-stable-api-investigation branch July 24, 2025 16:13

mashhurs mentioned this pull request Jul 24, 2025

[Move to ES Bridge -> plugin scope]: Move classes to Bridge API elastic/logstash-filter-elastic_integration#333

Open

mashhurs linked an issue Aug 8, 2025 that may be closed by this pull request

[Move to ES bridge -> ES scope] Introduce Bridge and proxies for GeoIP elastic/logstash-filter-elastic_integration#334

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. #131486

[Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. #131486

Uh oh!

mashhurs commented Jul 17, 2025

Uh oh!

elasticsearchmachine commented Jul 21, 2025

Uh oh!

yaauie left a comment

Uh oh!

Uh oh!

rjernst left a comment

Uh oh!

Uh oh!

Uh oh!

mashhurs commented Jul 22, 2025

Uh oh!

rjernst left a comment

Uh oh!

Uh oh!

yaauie left a comment

Uh oh!

Uh oh!

Uh oh!

[Logstash] Move elastic_integration plugin usage to ES logstash-bridge. #131486

[Logstash] Move elastic_integration plugin usage to ES logstash-bridge. #131486

Uh oh!

Conversation

mashhurs commented Jul 17, 2025

Why we are doing this?

What this PR does?

Uh oh!

elasticsearchmachine commented Jul 21, 2025

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mashhurs commented Jul 22, 2025

Uh oh!

rjernst left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yaauie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

[Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. #131486

[Logstash] Move `elastic_integration` plugin usage to ES logstash-bridge. #131486