[Feature Request] ShadowJar should've a default JSON merger

[Master-Code-Programmer]

Shadow Version 7.1.2

Gradle Version 7.5.1

Expected Behavior

A merging-transformer for JSON should be part of ShadowJAR, since JSON is one of the topmost used file formats.

Actual Behavior

There is no JSON transformer.
Sure, you can add one, and I did. But it was cumbersome and time consuming (even I converted only the Groovy code to Kotlin from this Ticket: #685). It should be part of the package.

JsonTransformer in buildSrc

package yourPackage

import org.gradle.api.file.FileTreeElement
import org.gradle.api.logging.Logging
import org.gradle.api.tasks.Input
import org.gradle.api.tasks.Optional
import com.google.gson.JsonArray
import com.google.gson.JsonNull
import com.google.gson.JsonObject
import com.google.gson.JsonElement
import com.google.gson.JsonPrimitive
import com.google.gson.JsonParser
import com.google.gson.Gson
import java.io.IOException
import java.io.InputStreamReader
import com.github.jengelman.gradle.plugins.shadow.transformers.CacheableTransformer
import com.github.jengelman.gradle.plugins.shadow.transformers.Transformer
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext
import com.github.jengelman.gradle.plugins.shadow.transformers.TransformerContext.*
import shadow.org.apache.tools.zip.ZipEntry
import shadow.org.apache.tools.zip.ZipOutputStream

@CacheableTransformer
class JsonTransformer : Transformer {
    @Optional
    @Input
    var resource: String? = null

    private var json: JsonElement? = null

    override fun canTransformResource(element: FileTreeElement): Boolean {
        val path = element.relativePath.pathString
        return resource != null && resource.equals(path, ignoreCase = true)
    }

    override fun transform(context: TransformerContext) {
        val j: JsonElement
        j = try {
            JsonParser.parseReader(InputStreamReader(context.getIs(), "UTF-8"))
        } catch (e: Exception) {
            throw RuntimeException("error on processing json", e)
        }
        json = if (json == null) j else mergeJson(json, j, "")
    }

    override fun hasTransformedResource(): Boolean {
        return json != null
    }

    override fun modifyOutputStream(os: ZipOutputStream, preserveFileTimestamps: Boolean) {
        val entry: ZipEntry = ZipEntry(resource)
        entry.setTime(
            getEntryTimestamp(
                preserveFileTimestamps, entry.getTime()
            )
        )
        try {
            os.putNextEntry(entry)
            os.write(GSON.toJson(json).toByteArray())
        } catch (e: IOException) {
            throw RuntimeException(e)
        }
        json = null
    }

    override fun getName(): String {
        return "JSON Transformer"
    }

    companion object {
        val GSON = Gson()
        val LOGGER = Logging.getLogger(JsonTransformer::class.java)

        /**
         * <table>
         * <tr>
         * <td>`lhs`</td> <td>`rhs`</td> <td>`return`</td>
        </tr> *
         * <tr>
         * <td>Any</td> <td>`JsonNull`</td> <td>`lhs`</td>
        </tr> *
         * <tr>
         * <td>`JsonNull`</td> <td>Any</td> <td>`rhs`</td>
        </tr> *
         * <tr>
         * <td>`JsonArray`</td> <td>`JsonArray`</td> <td>concatenation</td>
        </tr> *
         * <tr>
         * <td>`JsonObject`</td> <td>`JsonObject`</td> <td>merge for each key</td>
        </tr> *
         * <tr>
         * <td>`JsonPrimitive`</td> <td>`JsonPrimitive`</td>
         * <td>return lhs if `lhs.equals(rhs)`, error otherwise</td>
        </tr> *
         * <tr>
         * <td colspan="2">Other</td> <td>error</td>
        </tr> *
        </table> *
         *
         * @param lhs
         * a `JsonElement`
         * @param rhs
         * a `JsonElement`
         * @param id
         * used for logging purpose only
         * @return the merged `JsonElement`
         */
        fun mergeJson(lhs: JsonElement?, rhs: JsonElement?, id: String): JsonElement? {
            return if (rhs == null || rhs is JsonNull) {
                lhs
            } else if (lhs == null || lhs is JsonNull) {
                rhs
            } else if (lhs is JsonArray && rhs is JsonArray) {
                mergeJsonArray(lhs as JsonArray?, rhs as JsonArray?, id)
            } else if (lhs is JsonObject && rhs is JsonObject) {
                mergeJsonObject(lhs, rhs, id)
            } else if (lhs is JsonPrimitive && rhs is JsonPrimitive) {
                mergeJsonPrimitive(lhs, rhs, id)
            } else {
                LOGGER.warn("conflicts for property {} detected, {} & {}", id, lhs.toString(), rhs.toString())
                lhs
            }
        }

        fun mergeJsonPrimitive(lhs: JsonPrimitive, rhs: JsonPrimitive, id: String?): JsonPrimitive {
            if (lhs != rhs) {
                LOGGER.warn("conflicts for property {} detected, {} & {}", id, lhs.toString(), rhs.toString())
            }
            return lhs
        }

        fun mergeJsonObject(lhs: JsonObject, rhs: JsonObject, id: String): JsonObject {
            val `object` = JsonObject()
            val properties: MutableSet<String> = HashSet()
            properties.addAll(lhs.keySet())
            properties.addAll(rhs.keySet())
            for (property in properties) {
                `object`.add(
                    property, mergeJson(
                        lhs[property], rhs[property], "$id:$property"
                    )
                )
            }
            return `object`
        }

        fun mergeJsonArray(lhs: JsonArray?, rhs: JsonArray?, id: String?): JsonArray {
            val array = JsonArray()
            array.addAll(lhs)
            array.addAll(rhs)
            return array
        }
    }
}

Using it

import myPackage.JsonTransformer

tasks.withType<ShadowJar> {
    transform(
        JsonTransformer::class.java,
        jsonTransform("META-INF/additional-spring-configuration-metadata.json")
    )
   ...

Util function

fun jsonTransform(path: String): Action<JsonTransformer> {
    return object : Action<JsonTransformer> {
        override fun execute(it: JsonTransformer) {
            it.resource = path
        }
    }
}

Isn't this a bit much for one of the widest used file formats?

With txt, properties and html you can just use append and it will be fine, but JSON must be merged in a far more sophisticated manner.

[johnrengelman]

Thanks for the suggesting, but I'm not going to extend the capabilities of the plugin any further. Shadow offers a robust API to add additional transformers, and I would be happy to include a section in the docs for the community to list their own plugins.

[unaimillan]

@johnrengelman Any updates on this?

It would be wonderful to see such Community Transformers List.

Currently I'm very interested to find (or publish my own) XMLMergeTransformer that could unite several XML files with similar inner sections.

[Master-Code-Programmer]

@johnrengelman Any updates on this?

It would be wonderful to see such Community Transformers List.

Currently I'm very interested to find (or publish my own) XMLMergeTransformer that could unite several XML files with similar inner sections.

I have a PR which includes a JSON Transformer: #773.
Why, if @johnrengelman rejected the single transformer? That PR is the only reasonable way to use Spring with this plugin, because of its many conflicting files (and surely other Multi-Package frameworks have this problem too) . These files must be merged, therefore a JSON-Transformer must be included. Otherwise, creating a fat JAR for Spring seems extremely time consuming and cumbersome, since Spring has so many colliding files, which ShadowJar even doesn't tell you about. I'm not the first one running into this problem, so I started from another PR, fixed and extended it.

This seems particularly problematic, because Spring is used so frequently.