feat: add FDv2 FileDataSource initializer #1010

joker23 · 2025-11-18T23:08:44Z

This PR will introduce file data source initializer for FDv2 that could be specified by adding in an optional intializerOption to the FDv2 datasource option. If this option is not specified then the behavior should not change from before this change.

Additionally:

I've added some more type definitions to align better with FDv2 as well as a change set builder to convert FDv1 scheme to FDv2.
added a new custom data source options mode that allows users to specify the ordering of their initializers and synchronizers

NOTE: will need to look into a problem where while the initializers are running in expected order, the client initialized event seems to be prematurely firing. Example with the following config:

const ldClient = init(sdkKey, {
  dataSystem: {
    dataSource: {
      dataSourceOptionsType: 'custom',
      initializers: [
      {
        type: 'file',
        paths: [path.resolve(__dirname, 'flagdatav1.json')]
      },
      {
        type: 'polling'
      },
    ],
      synchronizers: [
        {
          type: 'streaming',
          streamInitialReconnectDelay: 1
        }
      ],
    }
  }
});

STDOUT with the sample-feature flag set to false in LD:

fs start
fs stop
polling started
*** SDK successfully initialized!
*** The 'sample-feature' feature flag evaluates to true.
        ██
          ██
      ████████
         ███████
██ LAUNCHDARKLY █
         ███████
      ████████
          ██
        ██

*** Event triggered: update:sample-feature
polling stopped
*** The 'sample-feature' feature flag evaluates to false.
info: [LaunchDarkly] Opened LaunchDarkly stream connection

In the above run, we have the correct ordering where the fs initializer got data, but since it does not report a valid basis, the polling initializer is also being ran. But as you can see, the client reported initialized before the polling initializer was able to update the data to the real thing (but further down we can see that the flag value is being set correctly before the streaming synchronizer runs)

I think this issue could be addressed in a separate change.

Note

Introduce FDv2 file-based initializer and customizable initializer/synchronizer pipeline, plus an adaptor to convert FDv1 payloads to FDv2 and related typing updates.

FDv2 core:
- FDv1→FDv2 adaptor: Add FDv1PayloadAdaptor to build FDv2 change sets from FDv1 payloads; export via internal/fdv2/index.
- Proto/processor typing: Strengthen proto types (EventType, ObjectKind, etc.); allow empty state in payload-transferred and optional id.
SDK server:
- File initializer: Implement FileDataInitializerFDv2 to load flags/segments from JSON or YAML (optional parser) and emit FDv2 payloads.
- Polling FDv2: Use FDv1PayloadAdaptor to handle FDv1 responses (selector FDv1Fallback).
- Custom data source options: Add dataSourceOptionsType: 'custom' with ordered initializers (file/polling) and synchronizers (streaming/polling); wire into LDClientImpl.
- Config: Export DEFAULT_STREAM_RECONNECT_DELAY; validation accepts custom options.
Tests:
- Add comprehensive tests for FDv1PayloadAdaptor, FileDataInitializerFDv2, and CompositeDataSource multi-initializer behavior.

^{Written by Cursor Bugbot for commit 718b11a. This will update automatically on new commits. Configure here.}

this commit will introduce the implementation of FDv2 File initiator

github-actions · 2025-11-18T23:09:59Z

@launchdarkly/browser size report
This is the brotli compressed size of the ESM build.
Compressed size: 169118 bytes
Compressed size limit: 200000
Uncompressed size: 789399 bytes

github-actions · 2025-11-18T23:10:06Z

@launchdarkly/js-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 25833 bytes
Compressed size limit: 26000
Uncompressed size: 126555 bytes

github-actions · 2025-11-18T23:10:29Z

@launchdarkly/js-client-sdk-common size report
This is the brotli compressed size of the ESM build.
Compressed size: 17636 bytes
Compressed size limit: 20000
Uncompressed size: 90259 bytes

github-actions · 2025-11-18T23:10:32Z

@launchdarkly/js-client-sdk size report
This is the brotli compressed size of the ESM build.
Compressed size: 22168 bytes
Compressed size limit: 25000
Uncompressed size: 76564 bytes

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts

packages/shared/sdk-server/src/options/Configuration.ts

tanderson-ld · 2025-11-19T14:16:21Z

packages/shared/common/src/internal/fdv2/proto.ts

+    | DeleteObject
+    | PayloadTransferred
+    | GoodbyeObject
+    | ErrorObject;


If new values are added to the protocol, is the code well behaved? Same for nearby types.

yea payload changes will potentially break this, but that is probably true even if we didn't declare these types?

tanderson-ld · 2025-11-19T14:24:37Z

packages/shared/common/src/internal/fdv2/FDv2ChangeSetBuilder.ts

Double check logic in the processFDv1FlagsAndSegments in PollingProcessorFDv2 matches this and/or consider commonizing.

ah yea the logic is the same! Let me think of a way to merge the logic.

Now that I am thinking about this more, I don't think sdk-common is the right place to create this builder? I am not sure how FDv2 is going to show up for client side implementations so probably move this out to sdk-server then, if we find out that client implementation is similar, move it back out to sdk-common? @abarker-launchdarkly @kinyoklion thoughts?

Otherwise, I think the plan will be change this to a FDv1PayloadAdaptor instead of a builder.

If the data inside the objects is opaque, then the the code itself would likely be the same for both of them.

I am fine with it not being common until we need it though.

A note specifically for internal/common code, but code we intend to use client-side we likely want to avoid classes and do this with functions, as generally speaking they end up smaller when minified. (Or old school function that returns object).

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts

tanderson-ld · 2025-11-19T14:38:42Z

packages/shared/sdk-server/src/options/Configuration.ts

          'DataSourceOptions',
          typeof options.dataSource,
        ),
      ];


Should a customer be able to configure to use only a file data source? @kinyoklion thoughts?

I think they should be able to configure just the file data source with polling and streaming disabled, but you'll have to confirm with others on team as I haven't dealt with file data source implementations.

I think we discussed sync, but yeah it should be possible. This would be a typical use case with FDv1. Often times people use file data sources for testing purposes.

Okay I think we can have a separate story to handle file synchronizer? I think it would be a fairly trivial implementation on top of this PR.

packages/shared/sdk-server/src/LDClientImpl.ts

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts

this commit adds a fdv1 payload adaptor to process fdv1 data sets

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts

this enables users to specify file data source initializer

joker23 · 2025-11-21T00:07:32Z

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts

+            dataCallback(false, { initMetadata, payload });
+          });
+
+          statusCallback(subsystemCommon.DataSourceState.Valid);


@tanderson-ld what does marking the state as valid do exactly? Is this what is telling the client that this initializer is successful?

joker23 added 2 commits November 18, 2025 16:28

feat: add FDv2 file data source initiator

66c80c7

this commit will introduce the implementation of FDv2 File initiator

style: [sdk-1577] fixing lint issues

678e42f

joker23 requested a review from a team as a code owner November 18, 2025 23:08

joker23 changed the title ~~Szhang/sdk 1577/f dv2 file initializer~~ feat: add FDv2 FileDataSource initializer Nov 18, 2025

cursor bot reviewed Nov 18, 2025

View reviewed changes

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts Outdated Show resolved Hide resolved

packages/shared/sdk-server/src/options/Configuration.ts Outdated Show resolved Hide resolved

joker23 force-pushed the szhang/sdk-1577/FDv2-file-initializer branch 2 times, most recently from 5d8cadd to cf28ade Compare November 18, 2025 23:20

tanderson-ld reviewed Nov 19, 2025

View reviewed changes

test: [sdk-1577] add unit tests for file data initializer

15d65d7

joker23 force-pushed the szhang/sdk-1577/FDv2-file-initializer branch from cf28ade to 15d65d7 Compare November 19, 2025 15:25

cursor bot reviewed Nov 19, 2025

View reviewed changes

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts Show resolved Hide resolved

chore: addressing PR feedback

10363a9

joker23 force-pushed the szhang/sdk-1577/FDv2-file-initializer branch from 24bee72 to 10363a9 Compare November 19, 2025 21:42

joker23 commented Nov 19, 2025

View reviewed changes

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts Outdated Show resolved Hide resolved

joker23 added 2 commits November 20, 2025 14:25

feat: FDv1 payload adaptor

b02abe9

this commit adds a fdv1 payload adaptor to process fdv1 data sets

chore: revert datasource config and add custom datasource option

b1e0123

cursor bot reviewed Nov 20, 2025

View reviewed changes

packages/shared/sdk-server/src/data_sources/fileDataInitilizerFDv2.ts Show resolved Hide resolved

joker23 force-pushed the szhang/sdk-1577/FDv2-file-initializer branch 2 times, most recently from b1b1883 to e1b1349 Compare November 20, 2025 23:05

feat: implement the custom data source option

718b11a

this enables users to specify file data source initializer

joker23 force-pushed the szhang/sdk-1577/FDv2-file-initializer branch from e1b1349 to 718b11a Compare November 20, 2025 23:58

joker23 commented Nov 21, 2025

View reviewed changes

joker23 requested review from kinyoklion and tanderson-ld November 21, 2025 00:07

feat: add FDv2 FileDataSource initializer #1010

Are you sure you want to change the base?

feat: add FDv2 FileDataSource initializer #1010

Uh oh!

Conversation

joker23 commented Nov 18, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Nov 18, 2025

Uh oh!

github-actions bot commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joker23 Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

joker23 commented Nov 18, 2025 •

edited by cursor bot

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

github-actions bot commented Nov 18, 2025 •

edited

Loading

joker23 Nov 19, 2025 •

edited

Loading