Fix modified flag for RemovePermutesAroundElementwiseOps (pytorch#16047)

Summary:

This change fixes an issue where the `RemovePermutesAroundElementwiseOps` pass was not correctly returning the modified flag to indicate whether the graph was changed during the optimization.

The modified flag is crucial for the pass manager to determine if additional passes need to be run or if the optimization has reached a fixed point. Without properly returning this flag, the pass manager may not iterate correctly or may run unnecessary additional passes.

Differential Revision: D88109865

Author: eigen-k

backends/cadence/aot/remove_ops.py

@@ -566,13 +566,18 @@ def call(self, graph_module: torch.fx.GraphModule) -> PassResult:
                     for node in subgraph.nodes:
                         processed_nodes.add(node)
 
+        modified = False
         for subgraph in subgraphs_found:
             self.permute_subgraph(subgraph)
+            modified = True
 
-        graph_module.graph.eliminate_dead_code()
-        graph_module.recompile()
 
-        return super().call(graph_module)
+        if modified:
+            graph_module.graph.eliminate_dead_code()
+            graph_module.recompile()
+            return super().call(graph_module)
+
+        return PassResult(graph_module, False)
 
     def visit(
         self,