New fauna-shell only supports queries up to ~100KB

[findgriffin]

I think fauna supports query sizes up to several megabytes, so this is a shell/driver bug:

davidgriffin@skadi flotsam % fauna --version
4.0.0-beta
davidgriffin@skadi flotsam % cat big_97k_query.fql| fauna query --profile DevDemoDB -
{
  ...
}
davidgriffin@skadi flotsam % cat big_102k_query.fql| fauna query --profile DevDemoDB -
 ... help text ...

An unexpected error occurred...

EAGAIN: resource temporarily unavailable, read

If you believe this is a bug, please report this issue on GitHub: https://github.com/fauna/fauna-shell/issues

Running with --verbosity=9 the 102k query actually succeeds, but if I throw a 1MB file at it...

davidgriffin@skadi flotsam % cat big_1M_query.fql| fauna query --verbosity=9 --profile DevDemoDB -
... help text ...
An unexpected error occurred...

EAGAIN: resource temporarily unavailable, read

If you believe this is a bug, please report this issue on GitHub: https://github.com/fauna/fauna-shell/issues
[error]: EAGAIN: resource temporarily unavailable, read
    at readFileSync (node:fs:448:20)
    at resolveInput (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96814:12)
    at Object.queryCommand [as handler] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96824:22)
    at /usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80016:46
    at maybeAsyncResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79818:83)
    at CommandInstance.handleValidationAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80015:19)
    at CommandInstance.applyMiddlewareAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80046:242)
    at CommandInstance.runCommand (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79953:212)
    at [runYargsParserAndExecuteCommands] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82735:91)
    at YargsInstance.parse (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82072:59)

I'm loving the new CLI so keep up the good work!

[findgriffin]

Also, I'm on an arm64 Macbook pro with MacOS 15.

[findgriffin]

Full verbose=9 output...

davidgriffin@skadi flotsam % cat big_1M_query.fql| fauna query --verbosity=9 --profile DevDemoDB -
[config]: Found default config file named "fauna.config.yaml". Using it.
[config]: Reading config from fauna.config.yaml.
[config]: Using profile DevDemoDB...
[config]: Applying config: {
    "color": true,
    "secret": "REDACTED"
    "url": "https://db.fauna-dev.com"
}
[config]: Found default config file named "fauna.config.yaml". Using it.
[config]: Reading config from fauna.config.yaml.
[config]: Using profile DevDemoDB...
[config]: Applying config: {
    "color": true,
    "secret": "REDACTED",
    "url": "https://db.fauna-dev.com"
}
[argv]: {
    "_": [
        "query"
    ],
    "verbosity": 9,
    "profile": "DevDemoDB",
    "p": "DevDemoDB",
    "root": "/Users/davidgriffin/IdeaProjects",
    "ops": "/Users/davidgriffin/IdeaProjects/ops",
    "util": "/Users/davidgriffin/IdeaProjects/ops/util",
    "bin": "/Users/davidgriffin/IdeaProjects/ops/bin",
    "color": true,
    "secret": "fnAF********************************P9X7",
    "url": "https://db.fauna-dev.com",
    "config": ".",
    "json": false,
    "quiet": false,
    "verbose-component": [],
    "verboseComponent": [],
    "user": "default",
    "u": "default",
    "local": false,
    "api-version": "10",
    "v": "10",
    "apiVersion": "10",
    "format": "fql",
    "f": "fql",
    "timeout": 5000,
    "performance-hints": false,
    "performanceHints": false,
    "include": [
        "summary"
    ],
    "account-url": "https://account.fauna.com",
    "accountUrl": "https://account.fauna.com",
    "$0": "fauna",
    "fql": "-"
}
[argv]: Existing Fauna environment variables: {"FAUNA_ROOT":"/Users/davidgriffin/IdeaProjects","FAUNA_OPS":"/Users/davidgriffin/IdeaProjects/ops","FAUNA_UTIL":"/Users/davidgriffin/IdeaProjects/ops/util","FAUNA_BIN":"/Users/davidgriffin/IdeaProjects/ops/bin"}
[argv]: reading query from stdin
[error]: unknown error thrown: Error
[error]: EAGAIN: resource temporarily unavailable, read
    at readFileSync (node:fs:448:20)
    at resolveInput (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96814:12)
    at Object.queryCommand [as handler] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96824:22)
    at /usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80016:46
    at maybeAsyncResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79818:83)
    at CommandInstance.handleValidationAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80015:19)
    at CommandInstance.applyMiddlewareAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80046:242)
    at CommandInstance.runCommand (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79953:212)
    at [runYargsParserAndExecuteCommands] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82735:91)
    at YargsInstance.parse (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82072:59) {
  [stack]: [Getter/Setter],
  [message]: 'EAGAIN: resource temporarily unavailable, read',
  errno: -35,
  code: 'EAGAIN',
  syscall: 'read'
}
fauna query [fql]

Run an FQL query.

API:
  -u, --user               CLI user to run the command as. You must first log in as the user using 'fauna login'.                                                                        [string] [default: "default"]
      --local              Use a local Fauna container. Sets --url to 'http://0.0.0.0:8443' and --secret to 'secret'.                                                                       [boolean] [default: false]
      --url                URL for Core HTTP API requests made by the command. Defaults to https://db.fauna.com.                                                                                              [string]
      --secret             Secret used for authentication. Can't be used with --database or --role.                                                                                                           [string]
      --account-key        Fauna account key used for authentication. Can't be used with --user or --secret.                                                                                                  [string]
  -d, --database           Database, including Region Group and hierarchy, to run the command in. Ex: 'us/my_db', 'eu/parent_db/child_db', 'global/db'. Can't be used with --secret.                          [string]
  -r, --role               Role used to run the command. Can't be used with --secret.                                                                                                                         [string]
  -v, --api-version        FQL version to use.                                                                                                                           [string] [choices: "4", "10"] [default: "10"]
  -f, --format             Output format for the query. When present, --json takes precedence over --format. Only applies to v10 queries.                           [string] [choices: "fql", "json"] [default: "fql"]
      --typecheck          Enable typechecking. Defaults to the typechecking setting of the database.                                                                                                        [boolean]
      --timeout            Maximum query runtime in milliseconds. Only applies to v10 queries.                                                                                                [number] [default: 5000]
      --performance-hints  Output performance hints. Sets --include summary. Only applies to v10 queries. If no performance hints are returned, no hints are output.                        [boolean] [default: false]

Positionals:
  fql  FQL query to run. Use - to read from stdin.                                                                                                                                                            [string]

Output:
      --color  Enable color formatting. Use --no-color to disable.                                                                                                                           [boolean] [default: true]
      --json   Output the results as JSON.                                                                                                                                                  [boolean] [default: false]
      --quiet  Suppress all log messages except fatal errors. Overrides --verbosity and --verbose-component.                                                                                [boolean] [default: false]

Config:
      --config   Path to a CLI config file to use. If provided, you must specify a profile.                                                                                                    [string] [default: "."]
  -p, --profile  Profile from the CLI config file to use.                                                                                                                                                     [string]

Debug:
      --verbose-component  Components to emit logs for. Overrides the --verbosity flag. Pass values as a space-separated list. Ex: --verbose-component fetch error.
                                                                                                                            [array] [choices: "argv", "completion", "config", "creds", "error", "fetch"] [default: []]
      --verbosity          Least critical log level to emit. Accepts 1 (fatal) to 5 (debug). Lower values represent more critical logs.                                                          [number] [default: 0]

Options:
  -h, --help     Show help.                                                                                                                                                                                  [boolean]
      --version  Show the Fauna CLI version.                                                                                                                                                                 [boolean]
      --include  Query response info to output. Pass values as a space-separated list. Ex: --include summary queryTags.
                                                                                                    [array] [choices: "all", "none", "txnTs", "schemaVersion", "summary", "queryTags", "stats"] [default: ["summary"]]
  -i, --input    Path to a file containing an FQL query to run.                                                                                                                                               [string]
  -o, --output   Path to a file where query results are written. Defaults to stdout.                                                                                                                          [string]

Examples:
  fauna query "Collection.all()" --database us/my_db                               Run the query in the 'us/my_db' database and write the results to stdout.
  fauna query "Collection.all()" --database us/my_db --role server                 Run the query in the 'us/my_db' database using the 'server' role.
  fauna query "Collection.all()" --secret my-secret                                Run the query in the database scoped to a secret.
  fauna query -i /path/to/query.fql --database us/my_db                            Run the query from a file.
  echo "1 + 1" | fauna query - --database us/my_db                                 Run the query from stdin.
  fauna query -i /path/to/query.fql --output /tmp/result.json --database us/my_db  Run the query and write the results to a file.

An unexpected error occurred...

EAGAIN: resource temporarily unavailable, read

If you believe this is a bug, please report this issue on GitHub: https://github.com/fauna/fauna-shell/issues
[error]: EAGAIN: resource temporarily unavailable, read
    at readFileSync (node:fs:448:20)
    at resolveInput (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96814:12)
    at Object.queryCommand [as handler] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:96824:22)
    at /usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80016:46
    at maybeAsyncResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79818:83)
    at CommandInstance.handleValidationAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80015:19)
    at CommandInstance.applyMiddlewareAndGetResult (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:80046:242)
    at CommandInstance.runCommand (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:79953:212)
    at [runYargsParserAndExecuteCommands] (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82735:91)
    at YargsInstance.parse (/usr/local/lib/node_modules/fauna-shell/dist/cli.cjs:82072:59)

[ecooper]

This is the offending piece of code: https://github.com/fauna/fauna-shell/blob/main/src/commands/query.mjs#L71

Most likely, reading queries from stdin will probably need to use process.stdin.read()instead. We should consider upper limits to what is read into memory. And it all needs to be read into memory before it can be sent to Fauna.

[findgriffin]

This is the offending piece of code: https://github.com/fauna/fauna-shell/blob/main/src/commands/query.mjs#L71

Most likely, reading queries from stdin will probably need to use process.stdin.read()instead. We should consider upper limits to what is read into memory. And it all needs to be read into memory before it can be sent to Fauna.

Cool, it looks like the stated limit on the server side is 16MB, so I think that's what we want the driver/shell to support.