Change ENVs

[mgdenno]

teehr/src/teehr/evaluation/spark_session_utils.py

Line 522 in 885b709

if os.environ.get("IN_CLUSTER", "false").lower() == "true":

I think we should change this to use S3_ENDPOINT and S3_USE_PATH_STYLE_ACCESS to jive with other packages.

[mgdenno]

Hard coding this should be a red flag:

teehr/src/teehr/evaluation/spark_session_utils.py

Line 524 in 885b709

remote_configs[f"spark.sql.catalog.{remote_catalog_name}.s3.endpoint"] = "http://minio:9000"

[samlamont]

so just to clarify, this would be something like:

remote_configs[f"spark.sql.catalog.{remote_catalog_name}.s3.endpoint"] = os.environ.get(S3_ENDPOINT)
remote_configs[f"spark.sql.catalog.{remote_catalog_name}.s3.path-style-access"] = os.environ.get(S3_USE_PATH_STYLE_ACCESS)

which would eliminate the need to have:

if os.environ.get("IN_CLUSTER", "false").lower() == "true":

does that look right ?

[mgdenno]

@samlamont I think S3_USE_PATH_STYLE_ACCESS is a bool so would probably need that same treatment, but i think we can eliminate the IN_CLUSTER env in favor of S3_USE_PATH_STYLE_ACCESS which seems to be more standard and is really what we are changing. My bad for suggesting IN_CLUSTER in the first place.