Skip to content

Commit 9dfb0a8

Browse files
haraldschillyharaldschilly
committed
Merge remote-tracking branch 'origin/master' into github-tests-report-status
2 parents 45bb8e1 + 229b398 commit 9dfb0a8

File tree

343 files changed

+12886
-9780
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

343 files changed

+12886
-9780
lines changed

‎.github/workflows/make-and-test.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,7 @@ jobs:
106106
pip install ipykernel
107107
python -m ipykernel install --prefix=./jupyter-local --name python3-local --display-name "Python 3 (Local)"
108108
109+
109110
- name: install pnpm
110111
#run: cd src && npm install -g pnpm
111112
uses: pnpm/action-setup@v4
@@ -117,7 +118,7 @@ jobs:
117118
run: cd src && pnpm run make
118119

119120
- name: test
120-
run: source venv/bin/activate && cd src && pnpm run test:report
121+
run: source venv/bin/activate && cd src && pnpm run test-github-ci:report
121122

122123
- name: upload test results
123124
uses: actions/upload-artifact@v4 # upload test results

‎README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# CoCalc
22

3+
[![Make all packages and run their tests](https://github.com/sagemathinc/cocalc/actions/workflows/make-and-test.yml/badge.svg)](https://github.com/sagemathinc/cocalc/actions/workflows/make-and-test.yml)
4+
35
#### <u>_**Co**_</u>_llaborative_ <u>_**Calc**_</u>_ulation_
46

57
CoCalc is web-based software that enables collaboration in research, teaching, and scientific publishing. It includes [Jupyter Notebooks](https://cocalc.com/features/jupyter-notebook), [Sage Worksheets](https://cocalc.com/features/sage), a [LaTeX Editor](https://cocalc.com/features/latex-editor) and a [Linux Terminal](https://cocalc.com/features/terminal) to help people work together in real time from different locations. It also has a [Computational Whiteboard](https://cocalc.com/features/whiteboard) for expressing and sharing ideas and running code. It is available for free and [can be upgraded for internet access, better hosting quality, and other features](https://cocalc.com/store). It can also be used for [teaching courses](https://cocalc.com/features/teaching) with flexible [course license options](https://cocalc.com/pricing/courses). It is also possible to run CoCalc [on your own infrastructure](https://cocalc.com/pricing/onprem).

‎src/.claude/settings.local.json

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"permissions": {
3+
"allow": [
4+
"Bash(pnpm tsc:*)",
5+
"Bash(pnpm build:*)",
6+
"Bash(git add:*)",
7+
"Bash(git commit:*)"
8+
],
9+
"deny": []
10+
}
11+
}

‎src/CLAUDE.md

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
# CLAUDE.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
4+
5+
# CoCalc Source Repository
6+
7+
* This is the source code of CoCalc in a Git repository
8+
* It is a complex JavaScript/TypeScript SaaS application
9+
* CoCalc is organized as a monorepository (multi-packages) in the subdirectory "./packages"
10+
* The packages are managed as a pnpm workspace in "./packages/pnpm-workspace.yaml"
11+
12+
## Code Style
13+
14+
- Everything is written in TypeScript code
15+
- Indentation: 2-spaces
16+
- All .js and .ts files are formatted by the tool prettier
17+
- Add suitable types when you write code
18+
- Variable name styles are "camelCase" for local and "FOO_BAR" for global variables. If you edit older code not following these guidlines, adjust this rule to fit the files style.
19+
- Some older code is JavaScript or CoffeeScript, which will be translated to TypeScript
20+
- Use ES modules (import/export) syntax, not CommonJS (require)
21+
- Organize the list of imports in such a way: installed npm packages are on top, newline, then are imports from @cocalc's code base. Sorted alphabetically.
22+
23+
## Development Commands
24+
25+
### Essential Commands
26+
- `pnpm build-dev` - Build all packages for development
27+
- `pnpm clean` - Clean all node_modules and dist directories
28+
- `pnpm database` - Start PostgreSQL database server
29+
- `pnpm hub` - Start the main hub server
30+
- `pnpm psql` - Connect to the PostgreSQL database
31+
- `pnpm test` - Run full test suite
32+
- `pnpm test-parallel` - Run tests in parallel across packages
33+
- `pnpm depcheck` - Check for dependency issues
34+
35+
### Package-Specific Commands
36+
- `cd packages/[package] && pnpm tsc` - Watch TypeScript compilation for a specific package
37+
- `cd packages/[package] && pnpm test` - Run tests for a specific package
38+
- `cd packages/[package] && pnpm build` - Build a specific package
39+
40+
### Development Setup
41+
1. Start database: `pnpm database`
42+
2. Start hub: `pnpm hub`
43+
3. For TypeScript changes, run `pnpm tsc` in the relevant package directory
44+
45+
## Architecture Overview
46+
47+
### Package Structure
48+
CoCalc is organized as a monorepo with key packages:
49+
50+
- **frontend** - React/TypeScript frontend application using Redux-style stores and actions
51+
- **backend** - Node.js backend services and utilities
52+
- **hub** - Main server orchestrating the entire system
53+
- **database** - PostgreSQL database layer with queries and schema
54+
- **util** - Shared utilities and types used across packages
55+
- **comm** - Communication layer including WebSocket types
56+
- **conat** - CoCalc's container/compute orchestration system
57+
- **sync** - Real-time synchronization system for collaborative editing
58+
- **project** - Project-level services and management
59+
- **static** - Static assets and build configuration
60+
- **next** - Next.js server components
61+
62+
### Key Architectural Patterns
63+
64+
#### Frontend Architecture
65+
- **Redux-style State Management**: Uses custom stores and actions pattern (see `packages/frontend/app-framework/actions-and-stores.ts`)
66+
- **TypeScript React Components**: All frontend code is TypeScript with proper typing
67+
- **Modular Store System**: Each feature has its own store/actions (AccountStore, BillingStore, etc.)
68+
- **WebSocket Communication**: Real-time communication with backend via WebSocket messages
69+
70+
#### Backend Architecture
71+
- **PostgreSQL Database**: Primary data store with sophisticated querying system
72+
- **WebSocket Messaging**: Real-time communication between frontend and backend
73+
- **Conat System**: Container orchestration for compute servers
74+
- **Event-Driven Architecture**: Extensive use of EventEmitter patterns
75+
- **Microservice-like Packages**: Each package handles specific functionality
76+
77+
#### Communication Patterns
78+
- **WebSocket Messages**: Primary communication method (see `packages/comm/websocket/types.ts`)
79+
- **Database Queries**: Structured query system with typed interfaces
80+
- **Event Emitters**: Inter-service communication within backend
81+
- **REST-like APIs**: Some HTTP endpoints for specific operations
82+
83+
### Key Technologies
84+
- **TypeScript**: Primary language for all new code
85+
- **React**: Frontend framework
86+
- **PostgreSQL**: Database
87+
- **Node.js**: Backend runtime
88+
- **WebSockets**: Real-time communication
89+
- **pnpm**: Package manager and workspace management
90+
- **Jest**: Testing framework
91+
- **SASS**: CSS preprocessing
92+
93+
### Database Schema
94+
- Comprehensive schema in `packages/util/db-schema`
95+
- Query abstractions in `packages/database/postgres/`
96+
- Type-safe database operations with TypeScript interfaces
97+
98+
### Testing
99+
- **Jest**: Primary testing framework
100+
- **ts-jest**: TypeScript support for Jest
101+
- **jsdom**: Browser environment simulation for frontend tests
102+
- Test files use `.test.ts` or `.spec.ts` extensions
103+
- Each package has its own jest.config.js
104+
105+
### Import Patterns
106+
- Use absolute imports with `@cocalc/` prefix for cross-package imports
107+
- Example: `import { cmp } from "@cocalc/util/misc"`
108+
- Type imports: `import type { Foo } from "./bar"`
109+
- Destructure imports when possible
110+
111+
### Development Workflow
112+
1. Changes to TypeScript require compilation (`pnpm tsc` in relevant package)
113+
2. Database must be running before starting hub
114+
3. Hub coordinates all services and should be restarted after changes
115+
4. Use `pnpm clean && pnpm build-dev` when switching branches or after major changes
116+
117+
# Workflow
118+
- Be sure to typecheck when you're done making a series of code changes
119+
- Prefer running single tests, and not the whole test suite, for performance
120+
121+
## Git Workflow
122+
123+
- Prefix git commits with the package and general area. e.g. 'frontend/latex: ...' if it concerns latex editor changes in the packages/frontend/... code.
124+
- When pushing a new branch to Github, track it upstream. e.g. `git push --set-upstream origin feature-foo` for branch "feature-foo".
125+
126+
# important-instruction-reminders
127+
- Do what has been asked; nothing more, nothing less.
128+
- NEVER create files unless they're absolutely necessary for achieving your goal.
129+
- ALWAYS prefer editing an existing file to creating a new one.
130+
- NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.

‎src/CLAUDE.md.x

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,156 @@
1+
# CLAUDE.md and GEMINI.md
2+
3+
This file provides guidance to Claude Code (claude.ai/code) and also Gemini CLI (https://github.com/google-gemini/gemini-cli) when working with code in this repository.
4+
5+
# CoCalc Source Repository
6+
7+
- This is the source code of CoCalc in a Git repository
8+
- It is a complex JavaScript/TypeScript SaaS application
9+
- CoCalc is organized as a monorepository (multi-packages) in the subdirectory "./packages"
10+
- The packages are managed as a pnpm workspace in "./packages/pnpm-workspace.yaml"
11+
12+
## Code Style
13+
14+
- Everything is written in TypeScript code
15+
- Indentation: 2-spaces
16+
- Run `pretter -w [filename]` after modifying a file (ts, tsx, md, json, ...) to format it correctly.
17+
- All .js and .ts files are formatted by the tool prettier
18+
- Add suitable types when you write code
19+
- Variable name styles are "camelCase" for local and "FOO_BAR" for global variables. If you edit older code not following these guidlines, adjust this rule to fit the files style.
20+
- Some older code is JavaScript or CoffeeScript, which will be translated to TypeScript
21+
- Use ES modules (import/export) syntax, not CommonJS (require)
22+
- Organize the list of imports in such a way: installed npm packages are on top, newline, then are imports from @cocalc's code base. Sorted alphabetically.
23+
24+
## Development Commands
25+
26+
### Essential Commands
27+
28+
- `pnpm build-dev` - Build all packages for development
29+
- `pnpm clean` - Clean all node_modules and dist directories
30+
- `pnpm test` - Run full test suite
31+
- `pnpm depcheck` - Check for dependency issues
32+
- `prettier -w [filename]` to format the style of a file after editing it
33+
- after creating a file, run `git add [filename]` to start tracking it
34+
35+
### Package-Specific Commands
36+
37+
- `cd packages/[package] && pnpm build` - Build and compile a specific package
38+
- for packages/next and packages/static, run `cd packages/[package] && pnpm build-dev`
39+
- `cd packages/[package] && pnpm tsc:watch` - TypeScript compilation in watch mode for a specific package
40+
- `cd packages/[package] && pnpm test` - Run tests for a specific package
41+
- `cd packages/[package] && pnpm build` - Build a specific package
42+
- **IMPORTANT**: When modifying packages like `util` that other packages depend on, you must run `pnpm build` in the modified package before typechecking dependent packages
43+
44+
### Development
45+
46+
- **IMPORTANT**: Always run `prettier -w [filename]` immediately after editing any .ts, .tsx, .md, or .json file to ensure consistent styling
47+
- After TypeScript or `*.tsx` changes, run `pnpm build` in the relevant package directory
48+
49+
## Architecture Overview
50+
51+
### Package Structure
52+
53+
CoCalc is organized as a monorepo with key packages:
54+
55+
- **frontend** - React/TypeScript frontend application using Redux-style stores and actions
56+
- **backend** - Node.js backend services and utilities
57+
- **hub** - Main server orchestrating the entire system
58+
- **database** - PostgreSQL database layer with queries and schema
59+
- **util** - Shared utilities and types used across packages
60+
- **comm** - Communication layer including WebSocket types
61+
- **conat** - CoCalc's container/compute orchestration system
62+
- **sync** - Real-time synchronization system for collaborative editing
63+
- **project** - Project-level services and management
64+
- **static** - Static assets and build configuration
65+
- **next** - Next.js server components
66+
67+
### Key Architectural Patterns
68+
69+
#### Frontend Architecture
70+
71+
- **Redux-style State Management**: Uses custom stores and actions pattern (see `packages/frontend/app-framework/actions-and-stores.ts`)
72+
- **TypeScript React Components**: All frontend code is TypeScript with proper typing
73+
- **Modular Store System**: Each feature has its own store/actions (AccountStore, BillingStore, etc.)
74+
- **WebSocket Communication**: Real-time communication with backend via WebSocket messages
75+
76+
#### Backend Architecture
77+
78+
- **PostgreSQL Database**: Primary data store with sophisticated querying system
79+
- **WebSocket Messaging**: Real-time communication between frontend and backend
80+
- **Conat System**: Container orchestration for compute servers
81+
- **Event-Driven Architecture**: Extensive use of EventEmitter patterns
82+
- **Microservice-like Packages**: Each package handles specific functionality
83+
84+
#### Communication Patterns
85+
86+
- **WebSocket Messages**: Primary communication method (see `packages/comm/websocket/types.ts`)
87+
- **Database Queries**: Structured query system with typed interfaces
88+
- **Event Emitters**: Inter-service communication within backend
89+
- **REST-like APIs**: Some HTTP endpoints for specific operations
90+
- **API Schema**: API endpoints in `packages/next/pages/api/v2/` use Zod schemas in `packages/next/lib/api/schema/` for validation. These schemas must be kept in harmony with the TypeScript types sent from frontend applications using `apiPost` (in `packages/next/lib/api/post.ts`) or `api` (in `packages/frontend/client/api.ts`). When adding new fields to API requests, both the frontend types and the API schema validation must be updated.
91+
92+
### Key Technologies
93+
94+
- **TypeScript**: Primary language for all new code
95+
- **React**: Frontend framework
96+
- **PostgreSQL**: Database
97+
- **Node.js**: Backend runtime
98+
- **WebSockets**: Real-time communication
99+
- **pnpm**: Package manager and workspace management
100+
- **Jest**: Testing framework
101+
- **SASS**: CSS preprocessing
102+
103+
### Database Schema
104+
105+
- Comprehensive schema in `packages/util/db-schema`
106+
- Query abstractions in `packages/database/postgres/`
107+
- Type-safe database operations with TypeScript interfaces
108+
109+
### Testing
110+
111+
- **Jest**: Primary testing framework
112+
- **ts-jest**: TypeScript support for Jest
113+
- **jsdom**: Browser environment simulation for frontend tests
114+
- Test files use `.test.ts` or `.spec.ts` extensions
115+
- Each package has its own jest.config.js
116+
117+
### Import Patterns
118+
119+
- Use absolute imports with `@cocalc/` prefix for cross-package imports
120+
- Example: `import { cmp } from "@cocalc/util/misc"`
121+
- Type imports: `import type { Foo } from "./bar"`
122+
- Destructure imports when possible
123+
124+
### Development Workflow
125+
126+
1. Changes to TypeScript require compilation (`pnpm build` in relevant package)
127+
2. Database must be running before starting hub
128+
3. Hub coordinates all services and should be restarted after changes
129+
4. Use `pnpm clean && pnpm build-dev` when switching branches or after major changes
130+
131+
# Workflow
132+
133+
- Be sure to build when you're done making a series of code changes
134+
- Prefer running single tests, and not the whole test suite, for performance
135+
136+
## Git Workflow
137+
138+
- Never modify a file when in the `master` or `main` branch
139+
- All changes happen through feature branches, which are pushed as pull requests to GitHub
140+
- When creating a new file, run `git add [filename]` to track the file.
141+
- Prefix git commits with the package and general area. e.g. 'frontend/latex: ...' if it concerns latex editor changes in the packages/frontend/... code.
142+
- When pushing a new branch to Github, track it upstream. e.g. `git push --set-upstream origin feature-foo` for branch "feature-foo".
143+
144+
# Important Instruction Reminders
145+
146+
- Do what has been asked; nothing more, nothing less.
147+
- NEVER create files unless they're absolutely necessary for achieving your goal.
148+
- ALWAYS prefer editing an existing file to creating a new one.
149+
- REFUSE to modify files when the git repository is on the `master` or `main` branch.
150+
- NEVER proactively create documentation files (`*.md`) or README files. Only create documentation files if explicitly requested by the User.
151+
152+
# Ignore
153+
154+
- Ignore files covered by `.gitignore`
155+
- Ignore everything in `node_modules` or `dist` directories
156+
- Ignore all files not tracked by Git, unless they are newly created files

‎src/GEMINI.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
CLAUDE.md

‎src/package.json

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -15,14 +15,14 @@
1515
"database": "cd dev/project && ./start_postgres.py",
1616
"database-remove-locks": "./scripts/database-remove-locks",
1717
"c": "LOGS=/tmp/ DEBUG='cocalc:*' ./scripts/c",
18-
"version-check": "pip3 install --break-system-packages typing_extensions mypy || pip3 install --break-system-packages typing_extensions mypy && ./workspaces.py version-check && mypy scripts/check_npm_packages.py",
18+
"version-check": "pip3 install typing_extensions mypy || pip3 install --break-system-packages typing_extensions mypy && ./workspaces.py version-check && mypy scripts/check_npm_packages.py",
1919
"test-parallel": "unset DEBUG && pnpm run version-check && cd packages && pnpm run -r --parallel test",
20-
"test": "unset DEBUG && pnpm run version-check && cd packages && pnpm run -r test",
21-
"test:report": "unset DEBUG && pnpm run version-check && cd packages && pnpm run -r test:report",
20+
"test": "unset DEBUG && pnpm run depcheck && pnpm run version-check && ./workspaces.py test",
21+
"test-github-ci": "unset DEBUG && pnpm run depcheck && pnpm run version-check && ./workspaces.py test --exclude=jupyter,file-server --retries=1",
22+
"test-github-ci:report": "unset DEBUG && pnpm run depcheck && pnpm run version-check && ./workspaces.py test --exclude=jupyter,file-server --retries=1",
2223
"depcheck": "cd packages && pnpm run -r --parallel depcheck",
2324
"prettier-all": "cd packages/",
2425
"local-ci": "./scripts/ci.sh",
25-
"conat-server": "cd packages/server && pnpm conat-server",
2626
"conat-connections": "cd packages/backend && pnpm conat-connections",
2727
"conat-watch": "cd packages/backend && pnpm conat-watch",
2828
"conat-inventory": "cd packages/backend && pnpm conat-inventory"

‎src/packages/backend/bin/conat-connections.cjs

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,15 @@
11
const { conat } = require('@cocalc/backend/conat')
22
const { showUsersAndStats } = require('@cocalc/conat/monitor/tables');
33
const { conatServer } = require('@cocalc/backend/data')
4+
const { delay } = require("awaiting");
45

56
async function main() {
67
console.log("Connecting to", conatServer);
78
const maxMessages = process.argv[2] ? parseInt(process.argv[2]) : undefined;
89
const maxWait = process.argv[3] ? parseInt(process.argv[3]) : 3000;
910
const client = conat();
11+
await client.waitUntilSignedIn();
12+
await delay(1000);
1013
if(!maxMessages) {
1114
console.log("\nUsage: pnpm conat-connnections [num-servers] [max-wait-ms]\n")
1215
}

‎src/packages/backend/bin/conat-disconnect.cjs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,17 @@
11
const { conat } = require('@cocalc/backend/conat')
22
const { connections } = require('@cocalc/conat/monitor/tables');
3+
const { sysApi } = require("@cocalc/conat/core/sys");
4+
const { delay } = require("awaiting");
35

46
async function main() {
57
console.log("Disconnect Clients From Server")
68
const ids = process.argv.slice(2);
79
console.log(ids);
810
const client = await conat()
9-
await client.call('sys.conat.server').disconnect(ids);
11+
await client.waitUntilSignedIn();
12+
await delay(1000);
13+
const sys = sysApi(client);
14+
await sys.disconnect(ids);
1015
process.exit(0);
1116
}
1217

0 commit comments

Comments
 (0)