MINIFICPP-2650 - Processor c api #1987

adamdebreceni · 2025-07-08T15:33:28Z

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?

For code changes:

If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file?
If applicable, have you updated the NOTICE file?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

Copilot

Pull Request Overview

This PR introduces a comprehensive C API framework for MiniFi C++ that enables plugins to be written in C while leveraging the existing C++ infrastructure. The implementation adds a processor C API layer that allows C-based extensions to integrate seamlessly with the MiniFi processing pipeline.

Key changes include:

New C API interface with MiniFi-specific types and callback structures
C API framework implementation that bridges C extensions to C++ core functionality
Refactored include paths and modularized architecture to support multiple API layers
Enhanced extension loading system that can verify and load both C++ and C API extensions

Reviewed Changes

Copilot reviewed 102 out of 143 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
minifi-api/include/minifi-c/minifi-c.h	Complete C API interface definition with processor callbacks and data structures
libminifi/src/minifi-c.cpp	C API implementation that bridges C calls to C++ MiniFi functionality
core-framework/c-api-framework/	New framework providing C++ wrappers for C API components
libminifi/include/utils/CProcessor.h	C processor wrapper enabling C extensions to participate in C++ processing pipeline
core-framework/include/core/extension/Utils.h	Enhanced extension verification supporting both C++ and C API libraries

Comments suppressed due to low confidence (1)

libminifi/src/minifi-c.cpp:1

This reinterpret_cast is unsafe and could lead to undefined behavior. The function should verify the type safety or use a safer casting mechanism.

/**

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

core-framework/common/src/utils/Id.cpp

libminifi/src/minifi-c.cpp

extensions/python/types/PyScriptFlowFile.cpp

martinzink

Since #2029 is already merged could you please rebase and retarget this PR to main?

core-framework/c-api-framework/include/api/core/ProcessContext.h

core-framework/c-api-framework/include/api/core/ProcessorImpl.h

core-framework/c-api-framework/src/core/ProcessSession.cpp

szaszm

early API comments

minifi-api/include/minifi-c/minifi-c.h

szaszm · 2025-10-27T14:11:44Z

minifi-api/include/minifi-c/minifi-c.h

+  uint32_t dynamic_properties_count;
+  const MinifiDynamicProperty* dynamic_properties_ptr;


I think we're mixing description and runtime values here. In a processor description/manifest, dynamic properties are only represented as supported/unsupported, while their values are a runtime information. I would remove these members.

we need this for docs generation, as they are in a separate section

minifi-api/include/minifi-c/minifi-c.h

core-framework/c-api-framework/include/api/core/PublishedMetrics.h

minifi-api/include/minifi-c/minifi-c.h

core-framework/c-api-framework/include/api/core/logging/Logger.h

core-framework/c-api-framework/include/api/core/FlowFile.h

libminifi/src/minifi-c.cpp

core-framework/include/core/Relationship.h

core-framework/common/include/utils/gsl.h

core-framework/c-api-framework/include/api/utils/ProcessorConfigUtils.h

core-framework/c-api-framework/src/core/ProcessorImpl.cpp

extensions/llamacpp/processors/DefaultLlamaContext.cpp

extensions/llamacpp/tests/RunLlamaCppInferenceTests.cpp

extensions/python/ExecutePythonProcessor.cpp

libminifi/include/utils/CProcessor.h

libminifi/src/core/ProcessSession.cpp

libminifi/src/minifi-c.cpp

szaszm · 2025-11-17T14:43:56Z

minifi-api/include/minifi-cpp/core/ProcessorDescriptor.h


  virtual void setSupportedRelationships(std::span<const RelationshipDefinition> relationships) = 0;
  virtual void setSupportedProperties(std::span<const PropertyReference> properties) = 0;
+  virtual void setSupportedProperties(std::span<const Property> properties) = 0;


why is this necessary?

we previously had a Property::getReference that worked incorrectly discarding types and allowed_values, since despite its name a Property is not easy to convert to PropertyReference, the former stores std::vector<std::string> while the latter uses std::span<const std::string_view> (this caused an issue in python processors as well)

szaszm · 2025-11-26T11:14:15Z

cmake/Extensions.cmake

+        set_target_properties(${extension-name} PROPERTIES
+                RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+                ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+                WINDOWS_EXPORT_ALL_SYMBOLS TRUE)


I would add the export macros in the C API extensions and not export all symbols. CMake's GenerateExportHeader can make it simpler. At least that's the vision, but maybe there are blockers preventing this.

this only has utility in tests where we directly link the test to (in this case) libminifi-llamacpp.dll, since we are directly driving the processor in the test (not waiting for the extension to register the callbacks) we need access to all member function symbols

szaszm · 2025-11-26T11:24:21Z

extension-framework/cpp-extension-lib/include/api/core/logging/Logger.h

do we no longer log IDs? because I don't see them on the interface

the trimming + id append has been moved upstream into LoggerBase

minifi-api/include/minifi-c/minifi-c.h

adamdebreceni force-pushed the minifi-api-reduced branch from b3fc327 to 23cad31 Compare July 17, 2025 13:37

adamdebreceni force-pushed the processor-c-api branch from b4a4728 to 51d8a57 Compare August 8, 2025 14:39

adamdebreceni force-pushed the minifi-api-reduced branch from 6932d42 to 75cdd49 Compare August 11, 2025 14:23

adamdebreceni force-pushed the processor-c-api branch from fffbcd6 to d86c0ab Compare August 21, 2025 08:37

adamdebreceni changed the base branch from minifi-api-reduced to main August 21, 2025 09:54

adamdebreceni force-pushed the processor-c-api branch 2 times, most recently from f35808b to 9807e09 Compare September 10, 2025 11:57

adamdebreceni force-pushed the processor-c-api branch from b860117 to 0c0310a Compare September 22, 2025 11:13

adamdebreceni changed the base branch from main to MINIFICPP-2629 September 22, 2025 11:34

lordgamez force-pushed the MINIFICPP-2629 branch 3 times, most recently from 054fd16 to a57636f Compare October 1, 2025 08:28

adamdebreceni force-pushed the processor-c-api branch 2 times, most recently from bb60828 to 4de141c Compare October 7, 2025 12:45

adamdebreceni changed the title ~~WIP processor c api~~ MINIFICPP-2650 - Processor c api Oct 14, 2025

martinzink requested a review from Copilot October 15, 2025 08:45

Copilot AI reviewed Oct 15, 2025

View reviewed changes

core-framework/common/src/utils/Id.cpp Show resolved Hide resolved

libminifi/src/minifi-c.cpp Show resolved Hide resolved

extensions/python/types/PyScriptFlowFile.cpp Show resolved Hide resolved

adamdebreceni marked this pull request as ready for review October 15, 2025 14:59

martinzink reviewed Oct 20, 2025

View reviewed changes

core-framework/c-api-framework/include/api/core/ProcessContext.h Outdated Show resolved Hide resolved

core-framework/c-api-framework/include/api/core/ProcessorImpl.h Outdated Show resolved Hide resolved

core-framework/c-api-framework/src/core/ProcessSession.cpp Outdated Show resolved Hide resolved

adamdebreceni force-pushed the processor-c-api branch from 66b396a to 09ba849 Compare October 22, 2025 08:32

adamdebreceni changed the base branch from MINIFICPP-2629 to main October 22, 2025 08:32

adamdebreceni marked this pull request as draft October 22, 2025 12:04

adamdebreceni marked this pull request as ready for review October 22, 2025 12:04

adamdebreceni force-pushed the processor-c-api branch from 09ba849 to 0d54774 Compare October 27, 2025 13:20

szaszm reviewed Oct 27, 2025

View reviewed changes

martinzink added the priority Review these first label Oct 28, 2025

lordgamez reviewed Oct 28, 2025

View reviewed changes

core-framework/include/core/Relationship.h Outdated Show resolved Hide resolved

core-framework/common/include/utils/gsl.h Show resolved Hide resolved

core-framework/c-api-framework/include/api/utils/ProcessorConfigUtils.h Outdated Show resolved Hide resolved

lordgamez reviewed Oct 29, 2025

View reviewed changes

szaszm added the depends-on-another-PR label Oct 31, 2025

adamdebreceni added 14 commits November 24, 2025 16:03

MINIFICPP-2650 - Readd level query method

2e63846

MINIFICPP-2650 - Clang tidy fix

f101359

MINIFICPP-2650 - Rebase fix

40616dc

MINIFICPP-2650 - Win fix

b046a25

MINIFICPP-2650 - Review changes

e9dd092

MINIFICPP-2650 - Review changes

5b45168

MINIFICPP-2650 - Review changes

50ae439

MINIFICPP-2650 - Own strings, vectors in ClassDescription

cd13d65

MINIFICPP-2650 - Remove includes

87ae81e

MINIFICPP-2650 - Review changes

d6fae76

MINIFICPP-2650 - Review changes

992a58c

MINIFICPP-2650 - Revert

e6e9574

MINIFICPP-2650 - Review changes

c3cb41d

MINIFICPP-2650 - Add missing include

312f3b2

adamdebreceni force-pushed the processor-c-api branch from e4a6e0b to 312f3b2 Compare November 24, 2025 15:03

fgerlits approved these changes Nov 24, 2025

View reviewed changes

adamdebreceni added 6 commits November 24, 2025 16:26

MINIFICPP-2650 - Require api version for extension creation

6ac6709

MINIFICPP-2650 - Add comment, clang-tidy fix

cc64669

MINIFICPP-2650 - Linter fix

755a7c0

MINIFICPP-2650 - Rename cpp framework

174ce8b

MINIFICPP-2650 - Add missing include

a1e042f

MINIFICPP-2650 - Review changes

e47d479

szaszm reviewed Nov 26, 2025

View reviewed changes

martinzink reviewed Nov 26, 2025

View reviewed changes

minifi-api/include/minifi-c/minifi-c.h Outdated Show resolved Hide resolved

adamdebreceni added 2 commits November 27, 2025 09:19

MINIFICPP-2650 - Fix build

958dc2f

MINIFICPP-2650 - Remove unused

4af23b2

martinzink approved these changes Nov 27, 2025

View reviewed changes

lordgamez approved these changes Nov 27, 2025

View reviewed changes

martinzink closed this in ef3dd56 Dec 1, 2025

		uint32_t dynamic_properties_count;
		const MinifiDynamicProperty* dynamic_properties_ptr;

MINIFICPP-2650 - Processor c api #1987

MINIFICPP-2650 - Processor c api #1987

Uh oh!

Conversation

adamdebreceni commented Jul 8, 2025

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martinzink left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

szaszm left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

szaszm left a comment •

edited

Loading