Migration using NSS2CSS 12/2024

[bourgeoa]

• Some data :

  • 20241211
    accounts 62932 email 51834
    58326 from 2020-12-31
  • size : data + .db/users/users : 66.5 GB

• test migration from solidcommunity.net backup on 2024/12/15

  • step1 on initialization reading NSS accounts 10 mn
  • step 2 CSS accounts creation 1s per account => expected 16 hours
    • to investigate : Do not fully succeed. only copy 24805 pods (in 14 hours).
    • podname's account converted to lower case (NSS url use lower case also, but not the account)
  • step 3 : copy users folders/files : 1 hour (need verification)
  • step 4 : update password to NSS one (seems OK was able to login to bourgeoa and bourgeoa1)
  • step 5 : replace oidcIssuier in WebID seems OK (ending with '/')
  • step 6 : replace acl:defaultForNew with acl:default seems OK
  • step 7 : replace links : this broke at first account
  • issues
    • step 2
      • pod with name's including blank or @ sign cannot be created on CSS (not allowed in an email)
      • on account creation uppercase converted to lower case (email request ?)
      • only 24805 pods created out of 62932 pods
      • account created even if there is no data folder this induces ENOENT error in the steps 4 to 8
    • step 7 : replace links on files
      • pod name beginning with '+' (+devient) broke step 7 on code error
      • improve to be able re-run on all situations
      • actually only on .ttl .acl and .meta extensions
      • add .json and .jsonld extensions

	7️⃣  CSS: Update 24805 pods : replace links on files content
	file:///home/solid/NSS2CSS/copy-pods-to-css.mjs:292
			const patt = new RegExp(source)
						 ^

	SyntaxError: Invalid regular expression: /+deviant.solidcommunity.net/: Nothing to repeat
		at new RegExp (<anonymous>)
		at file:///home/solid/NSS2CSS/copy-pods-to-css.mjs:292:22

	Node.js v20.18.0

• can we reduce the migration duration.
  • Done by removing HTTP CRUD request to directly file create accounts
• after initial migration : migrating pods on request can be done on demand with no difficulty

[bourgeoa]

Temporarily Proposals : (do not create when there is an issue)

1. do not create account if the data folder do not exist
2. blank and @ in podName either do not copy or %encode --> temporarily do not create
3. do not create account with uppercase

[bourgeoa]

Preparing end January 2025 migration :

1. • disks

• solidcommunity.net data should be on disk 1
• disk 3 shall be for weekly backup of solidcommunity.net running instance
• disk 4 shall keep a solidcommunity.net data copy before migration

2. • install pivot

• npm install @solid/pivot@latest

# cd /usr/local/lib
# npm install @solid/pivot@latest   ## this is a local installation
# ls -al node_modules/@solid/pivot
# ls -al node_modules/css-mashlib
# ls -al node_modules/mashlib
# ls -al node_modules/@solid/community-server

# cd /usr/local/bin
# ln -s ../lib/node_modules/@solid/pivot pivot
# ln -s ../lib/node_modules/css-mashlib css-mashlib
# ln -s ../lib/node_modules/mashlib mashlib
# ln -s ../lib/node_modules/@solid/community-server community-server

• create and store dedicated customise-me.json

# cd /root/pivot-config
# cat /root/pivot-config/customise-solidcommunity.net.json
{
    "comment": "Basic overrides parameters for a production server",
    "@context": [
      "https://linkedsoftwaredependencies.org/bundles/npm/@solid/community-server/^7.0.0/components/context.jsonld",
      "https://linkedsoftwaredependencies.org/bundles/npm/@solid/pivot/^1.0.0/components/context.jsonld"
    ],
    "@graph": [
      {
        "comment": "The settings of your email server.",
        "@type": "Override",
        "overrideInstance": {
          "@id": "urn:solid-server:default:EmailSender"
        },
        "overrideParameters": {
          "@type": "BaseEmailSender",
          "senderName": "no-reply@solidcommunity.net",
          "emailConfig_host": "smtp.sendgrid.net",
          "emailConfig_port": 465,
          "emailConfig_auth_user": "apikey",
          "emailConfig_auth_pass": "xxxxxxxxx"
        }
      },
      {
        "comment": "The location of the new pod templates folder.",
        "@type": "Override",
        "overrideInstance": {
          "@id": "urn:solid-server:default:PodResourcesGenerator"
        },
        "overrideParameters": {
          "@type": "StaticFolderGenerator",
          "templateFolder": "/usr/local/bin/css-mashlib/templates/pod"
        }
      },
      {
        "comment": "Sets the maximum size of a single pod to 50MB.",
        "@type": "Override",
        "overrideInstance": {
            "@id": "urn:solid-server:default:QuotaStrategy"
        },
        "overrideParameters": {
            "@type": "PodQuotaStrategy",
            "limit_amount": 50000000,
            "limit_unit": "bytes"
        }
      },
      {
        "comment": "Serve Databrowser as default representation",
        "@id": "urn:solid-server:default:DefaultUiConverter",
        "@type": "ConstantConverter",
        "contentType": "text/html",
        "filePath": "/usr/local/bin/mashlib/dist/databrowser.html",
        "options_container": true,
        "options_document": true,
        "options_minQuality": 1,
        "options_enabledMediaRanges": [
          "text/turtle",
          "text/n3",
          "text/markdown",
          "application/ld+json",
          "application/json",
          "application/rdf+xml"
         ]
      },
      {
        "comment": "Serve Mashlib static files.",
        "@id": "urn:solid-server:default:StaticAssetHandler",
        "@type": "StaticAssetHandler",
        "assets": [
          {
            "@type": "StaticAssetEntry",
            "relativeUrl": "/browse.html",
            "filePath": "/usr/local/bin/mashlib/dist/browse.html"
          },
          {
            "@type": "StaticAssetEntry",
            "relativeUrl": "/mash.css",
            "filePath": "/usr/local/bin/mashlib/dist/mash.css"
          },
          {
            "@type": "StaticAssetEntry",
            "relativeUrl": "/mashlib.min.js",
            "filePath": "usr/local/bin/mashlib/dist/mashlib.min.js"
          },
          {
            "@type": "StaticAssetEntry",
            "relativeUrl": "/mashlib.min.js.map",
            "filePath": "/usr/local/bin/mashlib/dist/mashlib.min.js.map"
          }
        ]
      }
    ]
  }

• create pm2 script to run pivot on disk 1

# cat /root/pivot-config/pivot-solidcommunity.net.sh
npx community-solid-server \
 -c /usr/local/bin/pivot/config/prod.json /root/pivot-config/customise-solidcommunity.net.json \
 -f /mnt/volume_lon1_01/solidcommunity.net \
 -p 443 -b https://solidcommunity.net:443 \
 --httpsKey /etc/letsencrypt/live/solidcommunity.net/privkey.pem \
 --httpsCert /etc/letsencrypt/live/solidcommunity.net/fullchain.pem \
 -m /usr/local/bin/pivot

• create pm2 script to run pivot on disk 4 with port 8443

# cat /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
npx community-solid-server \
 -c /usr/local/bin/pivot/config/prod.json /root/pivot-config/customise-solidcommunity.net.json \
 -f /mnt/volume_lon1_04/solicommunity.net \
 -p 8443 -b https://solidcommunity.net:8443 \
 --httpsKey /etc/letsencrypt/live/solidcommunity.net/privkey.pem \
 --httpsCert /etc/letsencrypt/live/solidcommunity.net/fullchain.pem \
 -m /usr/local/bin/pivot

3. • last migration test from disk 3 to disk 4

• rm data on disk 4

# pm2 stop pivot-8443-mnt04
# rm -r /mnt/volume_lon1_04/test-subdomain-8443

• start PIVOT on disk 4 with pm2 script

# pm2 ls
# cat /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
# pm2 del pivot-8443-mnt04
# pm2 start --name pivot-8443-mnt04 /root/pivot-config/pivot-solidcommunity.net-8443-mnt04.sh
# pm2 ls
# pm2 logs pivot-8443-mnt04

• initialize server root

# cp -r /usr/local/bin/pivot/www /mnt/volume_lon1_04/solidcommunity.net
# ls -al /mnt/volume_lon1_04/solidcommunity.net/www/
.acl
index.html

• from NSS data on disk 3 to CSS on disk 4

# cd /home/solid/NSS2CSS
# cat solidcommunity.net-mnt04.sh
node copy-pods-to-css.mjs /home/solid/config-443-mnt03/config.json /mnt/volume_lon1_04/test-subdomain-8443/ https://solidcommunity.net:8443/ xxx@users.css.pod
# nohup ./solidcommunity.net-mnt04.sh > migration-results/nohup.out &
# jobs

4. • migrate NSS2CSS with source disk 4 to dest disk 1

• stop test on port 8443 with data on disk 4

# pm2 stop pivot-8443-mnt04

• rm data on disk 4

# rm -r /mnt/volume_lon1_04/test-subdomain-8443

• stop solidcommunity.net

# pm2 ls
# pm2 stop solid

• copy data from disk 1 to disk 4

# rsync -a /mnt/volume_lon1_01/solid.community/ /mnt/volume_lon1_04/solidcommunity.net.bak.nss/data --stats &
# rsync -a /mnt/volume_lon1_01/.db/oidc/users /mnt/volume_lon1_04/solidcommunity.net.bak.nss/.db/oidc/ --stats &
# ls -al /mnt/volume_lon1_04 | wc -l

• rm data on disk 1

# rm -r /mnt/volume_lon1_01/solid.community &
# rm -r /mnt/volume_lon1_01/.db/oidc/users &
# ls -al /mnt/volume_lon1_01/

• start PIVOT on disk 1 with pm2 script

# cat /root/pivot-config/pivot-solidcommunity.net.sh
# pm2 start --name pivot /root/pivot-config/pivot-solidcommunity.net.sh

• initialize server root

# cp -r /usr/local/bin/pivot/www /mnt/volume_lon1_01/solidcommunity.net
# ls -al /mnt/volume_lon1_01/solidcommunity.net/www/
.acl
index.html

• run migration script from source disk 4 to dest disk 1

# cd /home/solid/NSS2CSS
# nohup ./solidcommunity.net-mnt01.sh > migration-results/nohup.out &

• check migration results

# cat migration-results/nohup.out

• cleaning pm2

# pm2 del pivot-8443-mnt04
# pm2 del solid

5. • update backup cron job

# crontab -l
# crontab -e

• rm data on disk 3

# rm -r /mnt/*03/solidcommunity.net 

• run first backup
  rsync should exclude /www/.internal/idp

# rsync -a --dry-run  --exclude 'www/.internal/idp' /mnt/volume_lon1_01/solidcommunity.net/ /mnt/volume_lon1_03/solidcommunity.net.bak/ --stats > /var/log/solid/$(date +\%Y\%m\%d)-data.log
# rsync -a --exclude=www/.internal/idp --exclude=www/.internal/locks --exclude=www/.internal/accounts/cookies /mnt/volume_lon1_01/solidcommunity.net/ /mnt/volume_lon1_03/solidcommunity.net.bak/ --stats > /var/log/solid/$(date +\%Y\%m\%d)-data.log &
# ls -al /mnt/volume_lon1_03

• edit crontab

# crontab -e

6. • update wiki

[bourgeoa]

• NSS backup to disk 4
  • users # ls -al /mnt/volume_lon1_04/solidcommunity.net.bak/.db/o*/users/users | wc -l --> 65471
  • containers # ls -d /mnt/volume_lon1_04/solidcommunity.net.bak/data | wc -l --> 63111

root@solidcommunity:/mnt/volume_lon1_01#
Number of files: 2,418,646 (reg: 1,960,404, dir: 458,242)
Number of created files: 0
Number of deleted files: 0
Number of regular files transferred: 0
Total file size: 62,099,214,639 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 24,243,027
File list generation time: 3.169 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 81,434,137
Total bytes received: 471,469

sent 81,434,137 bytes  received 471,469 bytes  111,664.08 bytes/sec
total size is 62,099,214,639  speedup is 758.18

[bourgeoa]

• migration to CSS

NSS userFiles 65468

Invalid NSS config
	invalid Json	4
	deprecated config filename	3309
	invalid config keys	2293

invalid NSS pods 
	username with dot 1213
	no data folder 89
	no profile/card 92
	only solid.community webId 40652
	external webId 6
	username with arobase 0
	username with blank 0
	username with uppercase letter 1097

valid NSS pods 16713
	check control (should be zero)	0

CSS pods
	already existing CSS pods 2 (doctorbud, cdr)
	created CSS pods 16713
	failed create CSS pods 0
	failed CSS pod fetch 12624
	check control (should be zero)	-12626

oidcIssuer issue 33
	asymmetriciris
	bramb
	cacao
	dietercasier
	edwin
	espy
	ewout
	joshuaprout2
	kenji
	kludgy
	liamthys
	liuwei
	lucasdb99
	mrchanmp
	nbrooks22
	ordina
	pferreir
	psycop
	reza-soltani
	ruben-greeter
	solid-test-1
	solidtestsuite
	soltanireza65
	tbaly
	teamlion
	test002
	test003
	testkraken
	testkraken2
	tmey
	tradstuc
	unitest
	vanesstalkers

CSS failed pod fetch
littlestone	https://liitlestone.solidcommunity.net/card#me
larrygeyer77	https//larrygeyer77.solidcommunity.net/profile/card#me
solid-arne-test	https://vandoorslaerarne.solidcommunity.net/profile/card#me