Skip to content

Conversation

@VikaCep
Copy link
Contributor

@VikaCep VikaCep commented Nov 20, 2025

Ref https://github.com/grafana/support-escalations/issues/19505
Related to #911

Support custom Loki datasources in Synthetic Monitoring

Problem

The Synthetic Monitoring app hardcoded fallbacks to uid: 'grafanacloud-logs' for log queries, preventing users from configuring alternative Loki datasources. This blocked use cases where customers need to use different Loki datasources.

Affected areas:

  • Runtime log queries always tried grafanacloud-logs UID first
  • Initialization logic prioritized hardcoded UIDs over provisioned configuration

Solution

Updated the datasource resolution logic to respect the configured datasource in jsonData.logs before falling back to hardcoded defaults.

Changes:

  1. DataSource.ts - getLogsDS() and getMetricsDS()

    • Now attempt to use the configured datasource from jsonData first
    • Only fall back to hardcoded Grafana Cloud UIDs if the configured datasource doesn't exist
  2. useAppInitializer.ts - findDatasourceByNameAndUid()

    • Updated to try finding datasources by configured UID before falling back to default Cloud UIDs
    • Respects datasource configuration from provisioning metadata

Testing

To test:

  1. Provision SM plugin with custom Loki datasource in jsonData.logs
  2. Verify runtime queries use the configured datasource
  3. Verify fallback behavior when configured datasource doesn't exist

Notes

GCOM Provisioning Required:
If the SM datasource is deleted and re-initialized, it will reset to whatever GCOM provisions (the default datasource).

Future Enhancement:
We may evaluate adding a UI dropdown in the plugin configuration to allow users to select from available Loki datasources, removing the dependency on GCOM provisioning for this use case.

image

@VikaCep VikaCep requested a review from ckbedwell November 20, 2025 13:51
@VikaCep VikaCep self-assigned this Nov 20, 2025
@VikaCep VikaCep requested a review from a team as a code owner November 20, 2025 13:51
@github-actions github-actions bot added the fix A fix applied to the application. label Nov 20, 2025
@github-actions
Copy link

github-actions bot commented Nov 20, 2025

Script size changes

Name +/- Main This PR Outcome
[502.js] = 1,736.41 kB 1,736.41 kB
[832.js] +0.01% 958.91 kB 959.03 kB
[datasource/module.js] -0.12% 24.49 kB 24.46 kB
[692.js] = 20.19 kB 20.19 kB
[663.js] = 5.84 kB 5.84 kB
[module.js] = 4.92 kB 4.92 kB

Totals

Name +/- Main This PR Outcome
[Scripts] = 2,750.77 kB 2,750.86 kB
[Non-script Assets] = 2,655.48 kB 2,655.48 kB
[All] = 5,406.25 kB 5,406.33 kB

Generated by 🚫 dangerJS against 8ed0447

@VikaCep VikaCep force-pushed the feat/choose-logs-datasource branch from 4620236 to 8ed0447 Compare November 20, 2025 14:00
@ka3de
Copy link
Contributor

ka3de commented Nov 25, 2025

If the SM datasource is deleted and re-initialized, it will reset to whatever GCOM provisions (the default datasource).

That sentence I think indicates that, if we want to support this, we probably need to do it at a higher level.
The current situation is that Hosted Grafana presets the Hosted Metrics and Hosted Logs as the metrics and logs destinations for Synthetic Monitoring. There are some assumptions that count on this, for example in the plugin, which is what this PR tries to change, but also in the backend, for example alerts expect to find the metrics through the grafanacloud-prom datasource. The backend actually enforces the usage of the Hosted Metrics and Hosted Logs instances by comparing supplied data in the registration endpoint with the data in GCOM.

If we want to support this, we probably need to allow the user to specify the data source configuration, that will have to be stored in backend, and the plugin should consume whatever is set there. Additionally this should be propagated to HG somehow, so resetting the plugin does not revert the configuration.
This probably deserves a better discussion. I would consider creating an issue for it so we can triage it.

@VikaCep
Copy link
Contributor Author

VikaCep commented Nov 25, 2025

If we want to support this, we probably need to allow the user to specify the data source configuration, that will have to be stored in backend, and the plugin should consume whatever is set there. Additionally this should be propagated to HG somehow, so resetting the plugin does not revert the configuration.
This probably deserves a better discussion. I would consider creating an issue for it so we can triage it.

I agree, this PR was aimed specifically to address the use case of https://github.com/grafana/support-escalations/issues/19505. They were previously taking advantage of a workaround in order to consume the LBAC datasource and it stopped working, so this solution was a temporary fix. (see https://raintank-corp.slack.com/archives/C0162C1K9E1/p1763384093931469)

I do agree the right and long-term solution is to explicitly allow users to select the desired datasources in the UI and store it, so it doesn't reset when the app is reinstalled or the SM datasource restored, but as you said, that would have to be triaged and treated as a new feature. We can close this PR if we want to take that road.

@ar2pi
Copy link

ar2pi commented Dec 1, 2025

@VikaCep any update on the fix? From the perspective of anyone using Synthetics this is a regression and should be fixed. Long term improvement sounds good, but let's address restoring a working version first 🙏

@VikaCep
Copy link
Contributor Author

VikaCep commented Dec 1, 2025

@ar2pi this PR is still up for review, and while it does try address the customer’s immediate issue, there are some inconsistencies in other layers of the architecture that we should consider as well, as Daniel pointed out. The customer was relying on a workaround that we had previously confirmed was not something we officially support, so I think it’s debatable whether this qualifies as a regression:

In a past thread, some of you confirmed that there are no supported ways to change the datasource used by synthetics tests. The workaround mentioned in that thread no longer works for the customer.

@ar2pi
Copy link

ar2pi commented Dec 1, 2025

@VikaCep This absolutely counts as a regression from a user perspective. For context: Synthetics logs worked fine for users who rely on a LBAC datasource for logs. Now, it doesn't.

Could we think in terms of win-win here? Have the temporary fix asap, restore user satisfaction. Then think longer term.

Copy link
Contributor

@ckbedwell ckbedwell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's just do it. What's the worst that can happen? 😬

It'll be up to the user to manage this properly and we can look at supporting this more officially in the future.

@VikaCep VikaCep merged commit 16de336 into main Dec 1, 2025
25 checks passed
@VikaCep VikaCep deleted the feat/choose-logs-datasource branch December 1, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix A fix applied to the application.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants