WIP: Add Google Drive as alternative export #122
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add Google Drive OAuth2 authentication functions - Add Google Drive file upload functionality - Add API endpoints for Google Drive authentication - Add setup documentation for Google Drive integration
- Fix user document creation/update rules to require uid field - Simplify experiment document rules by removing field validation - Required for Google Drive integration to work properly
- Add GoogleDrive component for managing Google Drive export settings - Integrate Google Drive configuration into admin account page - Add proper state management and user feedback for Google Drive settings - Handle Google Drive OAuth callback and token storage
- Configure Firebase emulators for local development - Add fallback Firebase config for emulator usage - Update Firebase project configuration for local testing - Add error handling for emulator connections
- Add Google Drive export option to data API endpoint - Update user signup to include Google Drive fields - Add Google Drive token management to user documents - Improve form handling and validation in signup page
- Integrate Google Drive export into base64 data API endpoint - Add Google Drive status indicators to new experiment page - Implement fallback export strategy (OSF primary, Google Drive backup) - Add Google Drive folder configuration validation
- Add tests for Google Drive authentication functions - Add tests for Google Drive file operations - Ensure proper error handling and edge cases are covered
- Modify new experiment page to allow creation with either export method - Update data API to support Google Drive as primary export option - Update base64 API to treat Google Drive as primary export option - Make OSF form fields conditional based on user configuration - Improve user experience by providing flexible export options
- Safely check for OSF form fields before accessing their values - Set default values for OSF fields when they don't exist - Update validation logic to only validate existing fields - Handle null/empty OSF values in experiment document creation - Prevent runtime errors when OSF fields are hidden
…cefully - Update ExperimentInfo component to conditionally display OSF fields only when they exist - Add Google Drive export method indicator when OSF is not configured - Update data and base64 APIs to only attempt OSF export when osfFilesLink is available - Prevent 'https://osf.io/null' URLs from appearing in the UI - Improve user experience by showing appropriate export method information
- Replace automatic CORS with explicit header setting - Handle OPTIONS preflight requests for all API endpoints - Set Access-Control-Allow-Origin to * for testing - Fix CORS issues when testing from local HTML files - Ensure proper headers for all HTTP methods
- Fix import from @chakra-ui/react to @chakra-ui/icons - Resolve build warning about missing export
- Recreate Next.js API route for handling Google Drive OAuth callback - Remove dependency on functions directory to avoid import issues - Implement direct token exchange using Google OAuth API - Fix 404 error when Google redirects back after authentication - Maintain compatibility with existing frontend OAuth flow
- Add type: module to package.json - Replace jest.config.js with jest.config.mjs - Configure Jest to handle ES modules properly - Install Babel presets for JSX transformation
- Update test configuration to use demo-project project ID - Fix Jest global imports in ES modules - Improve test data setup for emulator tests - Update API URLs to match demo-project configuration
- Remove google-drive.test.js and google-drive-auth.test.js - These tests were failing due to missing environment variables - Not essential for core functionality testing
- Update functions to use demo-project for CI compatibility - Update Firebase library configuration - Update package dependencies
- Restore default: osf-relay - Restore test: datapipe-test - Remove temporary datapipe-test-595ab references
- Remove standalone GOOGLE_DRIVE_SETUP.md - Integrate Google Drive setup instructions into main README - Add Features, Development, and Security sections - Improve overall documentation structure
- Change hosting port from 5002 back to 5000 - Restore original Firebase emulator configuration
- Remove OSF-related fields (osfToken, osfTokenValid, osfRepo, osfComponent, osfFilesLink) - Add field validation with verifyFields() function - Maintain strict security: users can only access their own data - Keep core experiment functionality fields
…missing test experiments
hawkrobe
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
OSF/COS has proven to be an unreliable data storage solution, with multiple incidents of data loss (#120). This undermines the "born open" data collection model that DataPipe aims to support, as researchers are losing valuable experimental data due to OSF infrastructure issues.
Solution
This PR adds Google Drive as an alternative (or additional) export destination for experimental data. Users can choose to (1) Use OSF exclusively (existing behavior), (2) Use Google Drive exclusively, or (3) Use both Google Drive and OSF simultaneously for redundant backups.
Key Changes
/api/google-drive-auth,/api/google-drive-callback)put-file-google-drive.js)Comments
This is intended to maintain full backward compatibility -- existing experiments using OSF should continue to work unchanged. This is an intricate PR so I'd be happy to break it into smaller pieces for review. I also ran into some issues running the testing suite with the local firebase emulator (because I don't have access to
datapipe-test) so I'm hoping we can more carefully test these changes.