Skip to content

Commit 84896ea

Browse files
asithadeclaude
authored and
mauriciozanettisalomao
committed
feat(backend): add snowflake data warehouse integration (#129)
* feat(backend): add snowflake data warehouse integration (LFXV2-671) - Add SnowflakeService with connection pooling, query execution, and caching - Implement LockManager for distributed operations and resource coordination - Add Snowflake-related interfaces, enums, and constants to shared package - Add comprehensive architecture documentation for Snowflake integration - Configure environment variables for Snowflake connection - Add @snowflake-sdk/client dependency This integration enables server-side data warehouse queries with proper connection management, query optimization, and distributed locking. Signed-off-by: Asitha de Silva <[email protected]> * feat(analytics): implement snowflake integration with error handling - Add AnalyticsController with three endpoints: * GET /api/analytics/active-weeks-streak (52 weeks of activity data) * GET /api/analytics/pull-requests-merged (last 30 days) * GET /api/analytics/code-commits (last 30 days) - Implement proper error handling: * AuthenticationError (401) for missing user email * ResourceNotFoundError (404) for no analytics data * Consistent error propagation via next(error) - Use SQL window functions for accurate aggregations: * SUM(DAILY_COUNT) OVER () for totals across filtered results * DATEADD(DAY, -30, CURRENT_DATE()) for 30-day filtering - Add analytics data interfaces to shared package: * ActiveWeeksStreakRow/Response * UserPullRequestsRow/Response * UserCodeCommitsRow/Response - Update frontend dashboard with Chart.js visualizations: * Interactive tooltips with formatted dates and counts * Real-time data from Snowflake via analytics service - Update Snowflake integration documentation: * Focus on service architecture and best practices * Add examples for parameterized queries and lazy initialization * Document error handling patterns for callers - Update .env.example with analytics endpoint usage comments LFXV2-671 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> Signed-off-by: Asitha de Silva <[email protected]> * feat(auth): add authelia support and refactor profile to use nats - Add Authelia OIDC claims middleware for Auth0 compatibility - Transform standard OIDC claims to Auth0 format - Only activates for non-Auth0 users (issuer check) - Maps preferred_username, splits name into given/family - Modifies user object in-place (not replaceable) - Refactor profile controller to use NATS exclusively - Remove Supabase dependency for user profile data - Construct UserProfile from OIDC token claims - Use NATS as sole authoritative source for metadata - Email management methods still use Supabase - Change user metadata field from 'title' to 'job_title' - Update UserMetadata interface - Update frontend components (profile edit, layout, header) - Update backend validation logic - Ensure consistent field naming across all layers - Clean up debug logging - Remove console.log statement from server.ts - Simplify authelia userinfo fetch logic - Minor improvements - Remove unused 'open' dependency from angular.json - Fix header component formatting - Optimize auth-helper to use OIDC data directly Generated with [Claude Code](https://claude.ai/code) Signed-off-by: Asitha de Silva <[email protected]> * build: configure snowflake sdk logging and add to gitignore - Set Snowflake SDK log level to ERROR by default - Add SNOWFLAKE_LOG_LEVEL environment variable support - Prevent snowflake.log from being committed to repository Signed-off-by: Asitha de Silva <[email protected]> * fix(server): remove middleware Signed-off-by: Asitha de Silva <[email protected]> --------- Signed-off-by: Asitha de Silva <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 4fb0d1e commit 84896ea

35 files changed

+5426
-329
lines changed

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,9 @@ npm-debug.log*
3636
yarn-debug.log*
3737
yarn-error.log*
3838

39+
# Snowflake SDK logs
40+
snowflake.log
41+
3942
# Misc
4043
.DS_Store
4144
*.pem

‎apps/lfx-one/.env.example‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,37 @@ SUPABASE_STORAGE_BUCKET=your-supabase-bucket-name
3030
# Internal k8s service DNS for NATS cluster
3131
NATS_URL=nats://lfx-platform-nats.lfx.svc.cluster.local:4222
3232

33+
############### SNOWFLAKE CONFIG ###############
34+
# Snowflake Connection Configuration
35+
# Used for analytics endpoints: active-weeks-streak, pull-requests-merged, code-commits
36+
# Account identifier in format: orgname-accountname (e.g., jnmhvwd-xpb85243)
37+
SNOWFLAKE_ACCOUNT=your-org-account
38+
# Service user for read-only analytical queries
39+
SNOWFLAKE_USERNAME=your-username
40+
# Warehouse for query execution
41+
SNOWFLAKE_WAREHOUSE=your-warehouse
42+
# Analytics database name
43+
SNOWFLAKE_DATABASE=your-database
44+
# User role with SELECT-only permissions
45+
SNOWFLAKE_ROLE=your-read-role
46+
47+
# Snowflake Authentication (choose one method)
48+
# Method 1: Direct API Key (recommended for containers/Docker)
49+
# SNOWFLAKE_API_KEY=your-private-key-here
50+
51+
# Method 2: Private Key File (recommended for local development)
52+
# Place rsa_key.p8 file in this directory (same location as .env)
53+
# The service will automatically detect and use it
54+
# SNOWFLAKE_PRIVATE_KEY_PASSPHRASE=your-passphrase-here
55+
56+
# Optional: Connection Pool Configuration (defaults shown)
57+
# SNOWFLAKE_MIN_CONNECTIONS=2
58+
# SNOWFLAKE_MAX_CONNECTIONS=10
59+
60+
# Optional: Lock Strategy for Query Deduplication (defaults to 'memory')
61+
# SNOWFLAKE_LOCK_STRATEGY=memory
62+
############### END SNOWFLAKE CONFIG ###############
63+
3364
# AI Service Configuration
3465
# OpenAI-compatible proxy for meeting agenda generation
3566
AI_PROXY_URL=https://litellm.tools.lfx.dev/chat/completions

‎apps/lfx-one/.gitignore‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -48,3 +48,10 @@ Thumbs.db
4848
/playwright/.auth/
4949
/playwright-report/
5050
/test-results/
51+
52+
# Snowflake
53+
rsa_key.p8
54+
rsa_key.pub
55+
56+
# Logs
57+
snowflake.log

‎apps/lfx-one/angular.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,9 @@
5151
"outputMode": "server",
5252
"ssr": {
5353
"entry": "src/server/server.ts"
54-
}
54+
},
55+
"allowedCommonJsDependencies": ["@linuxfoundation/lfx-ui-core"],
56+
"externalDependencies": ["snowflake-sdk"]
5557
},
5658
"configurations": {
5759
"production": {

‎apps/lfx-one/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@
5050
"pino-http": "^10.5.0",
5151
"primeng": "^19.1.4",
5252
"rxjs": "~7.8.2",
53+
"snowflake-sdk": "^2.3.1",
5354
"tslib": "^2.8.1"
5455
},
5556
"devDependencies": {

‎apps/lfx-one/src/app/app.component.ts‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ import { AuthContext } from '@lfx-one/shared/interfaces';
88
import { ToastModule } from 'primeng/toast';
99

1010
import { HeaderComponent } from './shared/components/header/header.component';
11-
import { AnalyticsService } from './shared/services/analytics.service';
11+
import { SegmentService } from './shared/services/segment.service';
1212
import { UserService } from './shared/services/user.service';
1313

1414
@Component({
@@ -19,15 +19,15 @@ import { UserService } from './shared/services/user.service';
1919
})
2020
export class AppComponent {
2121
private readonly userService = inject(UserService);
22-
private readonly analyticsService = inject(AnalyticsService);
22+
private readonly segmentService = inject(SegmentService);
2323

2424
public auth: AuthContext | undefined;
2525
public transferState = inject(TransferState);
2626
public serverKey = makeStateKey<AuthContext>('auth');
2727

2828
public constructor() {
29-
// Initialize Segment analytics
30-
this.analyticsService.initialize();
29+
// Initialize Segment tracking
30+
this.segmentService.initialize();
3131

3232
const reqContext = inject(REQUEST_CONTEXT, { optional: true }) as {
3333
auth: AuthContext;
@@ -51,8 +51,8 @@ export class AppComponent {
5151
this.userService.authenticated.set(true);
5252
this.userService.user.set(this.auth.user);
5353

54-
// Identify user with Segment analytics (pass entire Auth0 user object)
55-
this.analyticsService.identifyUser(this.auth.user);
54+
// Identify user with Segment tracking (pass entire Auth0 user object)
55+
this.segmentService.identifyUser(this.auth.user);
5656
}
5757
}
5858
}

‎apps/lfx-one/src/app/layouts/profile-layout/profile-layout.component.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ export class ProfileLayoutComponent {
145145
const profile = this.profile();
146146
if (!profile?.profile) return '';
147147

148-
return profile.profile.title || '';
148+
return profile.profile.job_title || '';
149149
});
150150
}
151151

‎apps/lfx-one/src/app/modules/dashboards/components/my-meetings/my-meetings.component.html‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,8 @@ <h2 class="font-display font-semibold text-gray-900">My Meetings</h2>
1717
</div>
1818

1919
<!-- Scrollable Content -->
20-
<div class="flex flex-col flex-1 max-h-[28.125rem] overflow-y-auto">
21-
<div class="flex flex-col gap-6" data-testid="dashboard-my-meetings-list">
20+
<div class="flex flex-col flex-1">
21+
<div class="flex flex-col gap-6 overflow-scroll max-h-[30rem]" data-testid="dashboard-my-meetings-list">
2222
@if (todayMeetings().length > 0 || upcomingMeetings().length > 0) {
2323
<!-- TODAY Section - only show if there are meetings today -->
2424
@if (todayMeetings().length > 0) {

‎apps/lfx-one/src/app/modules/dashboards/components/recent-progress/recent-progress.component.ts‎

Lines changed: 219 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,13 @@
33

44
import { CommonModule } from '@angular/common';
55
import { Component, computed, ElementRef, inject, ViewChild } from '@angular/core';
6+
import { toSignal } from '@angular/core/rxjs-interop';
7+
import { AnalyticsService } from '@app/shared/services/analytics.service';
68
import { PersonaService } from '@app/shared/services/persona.service';
79
import { ChartComponent } from '@components/chart/chart.component';
810
import { CORE_DEVELOPER_PROGRESS_METRICS, MAINTAINER_PROGRESS_METRICS } from '@lfx-one/shared/constants';
911

10-
import type { ProgressItemWithChart } from '@lfx-one/shared/interfaces';
12+
import type { ActiveWeeksStreakResponse, ProgressItemWithChart, UserCodeCommitsResponse, UserPullRequestsResponse } from '@lfx-one/shared/interfaces';
1113

1214
@Component({
1315
selector: 'lfx-recent-progress',
@@ -20,20 +22,66 @@ export class RecentProgressComponent {
2022
@ViewChild('progressScroll') protected progressScrollContainer!: ElementRef;
2123

2224
private readonly personaService = inject(PersonaService);
25+
private readonly analyticsService = inject(AnalyticsService);
26+
27+
/**
28+
* Active weeks streak data from Snowflake
29+
*/
30+
private readonly activeWeeksStreakData = toSignal(this.analyticsService.getActiveWeeksStreak(), {
31+
initialValue: {
32+
data: [],
33+
currentStreak: 0,
34+
totalWeeks: 0,
35+
},
36+
});
37+
38+
/**
39+
* Pull requests merged data from Snowflake
40+
*/
41+
private readonly pullRequestsMergedData = toSignal(this.analyticsService.getPullRequestsMerged(), {
42+
initialValue: {
43+
data: [],
44+
totalPullRequests: 0,
45+
totalDays: 0,
46+
},
47+
});
48+
49+
/**
50+
* Code commits data from Snowflake
51+
*/
52+
private readonly codeCommitsData = toSignal(this.analyticsService.getCodeCommits(), {
53+
initialValue: {
54+
data: [],
55+
totalCommits: 0,
56+
totalDays: 0,
57+
},
58+
});
2359

2460
/**
2561
* Computed signal that returns progress metrics based on the current persona
62+
* Merges hardcoded metrics with real data from Snowflake
2663
*/
2764
protected readonly progressItems = computed<ProgressItemWithChart[]>(() => {
2865
const persona = this.personaService.currentPersona();
66+
const activeWeeksData = this.activeWeeksStreakData();
67+
const pullRequestsData = this.pullRequestsMergedData();
68+
const codeCommitsDataValue = this.codeCommitsData();
69+
70+
const baseMetrics = persona === 'maintainer' ? MAINTAINER_PROGRESS_METRICS : CORE_DEVELOPER_PROGRESS_METRICS;
2971

30-
switch (persona) {
31-
case 'maintainer':
32-
return MAINTAINER_PROGRESS_METRICS;
33-
case 'core-developer':
34-
default:
35-
return CORE_DEVELOPER_PROGRESS_METRICS;
36-
}
72+
// Replace metrics with real data if available
73+
return baseMetrics.map((metric) => {
74+
if (metric.label === 'Active Weeks Streak') {
75+
return this.transformActiveWeeksStreak(activeWeeksData);
76+
}
77+
if (metric.label === 'Pull Requests Merged') {
78+
return this.transformPullRequestsMerged(pullRequestsData);
79+
}
80+
if (metric.label === 'Code Commits') {
81+
return this.transformCodeCommits(codeCommitsDataValue);
82+
}
83+
return metric;
84+
});
3785
});
3886

3987
protected scrollLeft(): void {
@@ -45,4 +93,167 @@ export class RecentProgressComponent {
4593
const container = this.progressScrollContainer.nativeElement;
4694
container.scrollBy({ left: 300, behavior: 'smooth' });
4795
}
96+
97+
/**
98+
* Transform Active Weeks Streak API response to chart format
99+
* API returns data in ascending order by WEEKS_AGO (0, 1, 2, 3...)
100+
* Display as-is with newest week (week 0) on the left
101+
*/
102+
private transformActiveWeeksStreak(data: ActiveWeeksStreakResponse): ProgressItemWithChart {
103+
// Use data as-is: week 0 (newest) on the left, older weeks on the right
104+
const chartData = data.data;
105+
106+
return {
107+
label: 'Active Weeks Streak',
108+
value: data.currentStreak.toString(),
109+
trend: data.currentStreak > 0 ? 'up' : 'down',
110+
subtitle: 'Current streak',
111+
chartType: 'bar',
112+
chartData: {
113+
labels: chartData.map((row) => `Week ${row.WEEKS_AGO}`),
114+
datasets: [
115+
{
116+
data: chartData.map((row) => row.IS_ACTIVE),
117+
borderColor: '#10b981',
118+
backgroundColor: 'rgba(16, 185, 129, 0.1)',
119+
fill: true,
120+
tension: 0.4,
121+
borderWidth: 2,
122+
pointRadius: 0,
123+
},
124+
],
125+
},
126+
chartOptions: {
127+
responsive: true,
128+
maintainAspectRatio: false,
129+
plugins: {
130+
legend: { display: false },
131+
tooltip: {
132+
enabled: true,
133+
callbacks: {
134+
title: (context) => context[0].label,
135+
label: (context) => {
136+
const isActive = context.parsed.y === 1;
137+
return isActive ? 'Active' : 'Inactive';
138+
},
139+
},
140+
},
141+
},
142+
scales: {
143+
x: { display: false },
144+
y: { display: false, min: 0, max: 1 },
145+
},
146+
},
147+
};
148+
}
149+
150+
/**
151+
* Transform Pull Requests Merged API response to chart format
152+
* API returns data in ascending order by ACTIVITY_DATE
153+
* Display as-is with oldest date on the left, newest on the right
154+
*/
155+
private transformPullRequestsMerged(data: UserPullRequestsResponse): ProgressItemWithChart {
156+
const chartData = data.data;
157+
158+
return {
159+
label: 'Pull Requests Merged',
160+
value: data.totalPullRequests.toString(),
161+
trend: data.totalPullRequests > 0 ? 'up' : 'down',
162+
subtitle: 'Last 30 days',
163+
chartType: 'line',
164+
chartData: {
165+
labels: chartData.map((row) => row.ACTIVITY_DATE),
166+
datasets: [
167+
{
168+
data: chartData.map((row) => row.DAILY_COUNT),
169+
borderColor: '#0094FF',
170+
backgroundColor: 'rgba(0, 148, 255, 0.1)',
171+
fill: true,
172+
tension: 0,
173+
borderWidth: 2,
174+
pointRadius: 0,
175+
},
176+
],
177+
},
178+
chartOptions: {
179+
responsive: true,
180+
maintainAspectRatio: false,
181+
plugins: {
182+
legend: { display: false },
183+
tooltip: {
184+
enabled: true,
185+
callbacks: {
186+
title: (context) => {
187+
const date = new Date(context[0].label);
188+
return date.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' });
189+
},
190+
label: (context) => {
191+
const count = context.parsed.y;
192+
return `PRs Merged: ${count}`;
193+
},
194+
},
195+
},
196+
},
197+
scales: {
198+
x: { display: false },
199+
y: { display: false },
200+
},
201+
},
202+
};
203+
}
204+
205+
/**
206+
* Transform Code Commits API response to chart format
207+
* API returns data in ascending order by ACTIVITY_DATE
208+
* Display as-is with oldest date on the left, newest on the right
209+
*/
210+
private transformCodeCommits(data: UserCodeCommitsResponse): ProgressItemWithChart {
211+
const chartData = data.data;
212+
213+
return {
214+
label: 'Code Commits',
215+
value: data.totalCommits.toString(),
216+
trend: data.totalCommits > 0 ? 'up' : 'down',
217+
subtitle: 'Last 30 days',
218+
chartType: 'line',
219+
chartData: {
220+
labels: chartData.map((row) => row.ACTIVITY_DATE),
221+
datasets: [
222+
{
223+
data: chartData.map((row) => row.DAILY_COUNT),
224+
borderColor: '#0094FF',
225+
backgroundColor: 'rgba(0, 148, 255, 0.1)',
226+
fill: true,
227+
tension: 0.4,
228+
borderWidth: 2,
229+
pointRadius: 0,
230+
},
231+
],
232+
},
233+
chartOptions: {
234+
responsive: true,
235+
maintainAspectRatio: false,
236+
plugins: {
237+
legend: { display: false },
238+
tooltip: {
239+
enabled: true,
240+
callbacks: {
241+
title: (context) => {
242+
const date = new Date(context[0].label);
243+
return date.toLocaleDateString('en-US', { month: 'short', day: 'numeric', year: 'numeric' });
244+
},
245+
label: (context) => {
246+
const count = context.parsed.y;
247+
return `Commits: ${count}`;
248+
},
249+
},
250+
},
251+
},
252+
scales: {
253+
x: { display: false },
254+
y: { display: false },
255+
},
256+
},
257+
};
258+
}
48259
}

0 commit comments

Comments
 (0)