Feature suggestion: Simplify JSON structure, reduce number of unique fields

[meltingscales]

Context: I'm using PowerBI to import Snaffler JSON output, and noticed some things that made it difficult to import and report off of.

Basically, I get 40 or so JSON keys, each duplicated 4 times when importing in PowerBI:

eventProperties.Black.FileResult.FileInfo.LastAccessTime
eventProperties.Black.FileResult.FileInfo.Extension
eventProperties.Black.FileResult.RwStatus.CanRead
eventProperties.Black.FileResult.MatchedRule.RuleName
eventProperties.Black.FileResult.FileInfo.FullName
(...)

eventProperties.Red.FileResult.FileInfo.LastAccessTime
eventProperties.Red.FileResult.FileInfo.Extension
eventProperties.Red.FileResult.RwStatus.CanRead
eventProperties.Red.FileResult.MatchedRule.RuleName
eventProperties.Red.FileResult.FileInfo.FullName
(...)

eventProperties.Yellow.FileResult.FileInfo.LastAccessTime
eventProperties.Yellow.FileResult.FileInfo.Extension
eventProperties.Yellow.FileResult.RwStatus.CanRead
eventProperties.Yellow.FileResult.MatchedRule.RuleName
eventProperties.Yellow.FileResult.FileInfo.FullName
(...)

eventProperties.Green.FileResult.FileInfo.LastAccessTime
eventProperties.Green.FileResult.FileInfo.Extension
eventProperties.Green.FileResult.RwStatus.CanRead
eventProperties.Green.FileResult.MatchedRule.RuleName
eventProperties.Green.FileResult.FileInfo.FullName
(...)

I propose the sub-key Black/Red/Yellow/Green be eliminated entirely, and flattened so that it's stored under eventProperties.Severity or something similar. It would make importing this JSON into tools like PowerBI much easier and reporting simpler.

Example:

{
  eventProperties: {
    severity: "Green",
    data: { (data goes here) }
  }
}

[meltingscales]

If there isn't a way to do this, I can easily create a post-scan processor that transforms and cleans the data in preparation for use with tools like PowerBI. I'd probably write it first in Python, then maybe as a sub-project in Snaffler.

[henrypost]

Here's a Python script that flattens the data and nests it under eventProperties.object:

import json
from typing import Dict, List

import dirtyjson

# CHANGEME File to process
input = "snaffler.json"

# CHANGEME output file path
output = "snaffler.flattened.json"

def fix_snaffler_json(snaffler_lines: List[str]) -> List[str]:

    fixed = ["["]

    i = 0
    for line in snaffler_lines:
        line = line.strip()
        if i >= len(snaffler_lines):
            pass  # No dangling comma
        else:
            fixed.append(line + "," + "\n")
        i += 1

    fixed.append("]")

    return fixed


def mutate_one(d: Dict) -> Dict:
    # make a deep copy of d
    new = json.loads(json.dumps(d))

    all_levels = list(new["eventProperties"].keys())
    assert (len(all_levels) == 1) or (
        len(all_levels) == 0
    )  # We should only have 1 or 0 level ever in an alert, at least that's what I've observed...

    if len(all_levels) == 1:
        level = all_levels[0]

        new["eventProperties"]["object"] = new["eventProperties"][level]
        new["eventProperties"]["object"]["level"] = level
        del new["eventProperties"][level]

    return new


def mutate_all(l: List[Dict]) -> List[Dict]:

    ret = []

    for d in l:
        ret.append(mutate_one(d))

    return ret


def read_snaffler_json_file(path: str) -> List[str]:
    with open(path, "r") as file:
        raw_data = file.readlines()

    return raw_data


if __name__ == "__main__":

    raw_data = read_snaffler_json_file(input)
    data = fix_snaffler_json(raw_data)
    data_str = "".join(data)

    data_json = dirtyjson.loads(data_str)

    mutated = mutate_all(data_json)

    with open(output, "w") as fh:
        json.dump(mutated, fh, indent=4)