Merge pull request #7 from haskell-perf/newStack

New stack

Author: nobrakal

.travis.yml

@@ -35,8 +35,8 @@ env:
 - DO=TIME STEP=0
 - DO=TIME STEP=1
 - DO=TIME STEP=2
+- DO=TIME STEP=3
 - DO=SPACE
-- DO=DATASIZE
 
 install:
 # Build dependencies
@@ -45,9 +45,8 @@ install:
 script:
 # Build the package, its tests, and its docs and run the tests
 - stack --no-terminal bench --no-run-benchmarks
-- PREFIX=.stack-work/dist/x86_64-linux/Cabal-2.0.1.0/build
-- if [ $DO = DATASIZE ]; then travis_wait $PREFIX/datasize/datasize; else if [ $DO = SPACE ]; then travis_wait 30 $PREFIX/space/space; else $PREFIX/time/time run --part $STEP --of 3  -g '("Mesh",2)' -g '("Clique",2)' ; fi; fi;
-
+- PREFIX=.stack-work/dist/x86_64-linux/Cabal-2.2.0.1/build
+- if [ $DO = SPACE ]; then travis_wait 30 $PREFIX/space/space; else $PREFIX/time/time run --part $STEP --of 4  -g '("Mesh",2)' -g '("Clique",2)' ; fi;
 
 before_install:
 # Download and unpack the stack executable

README.md

@@ -1,13 +1,14 @@
 # Benchmark suite for graph libraries
 
-## Warning: Under Active Developement
-This is being developed as part of the Google Summer of Code 2018.
-Please do not expect anything from the code for now. 
-
-Feel free to open an issue anyway :)
+## Intro
+This project was developed as part of the Google Summer of Code 2018.
+Feel free to open an issue :)
 
 ## Results
-Current results of `cabal bench time` and `cabal bench space` can be found here: https://travis-ci.org/haskell-perf/graphs
+
+![svg](https://raw.githubusercontent.com/haskell-perf/graphs/newStack/results/TIME.svg?sanitize=true)
+
+Current results of `cabal bench time` and `cabal bench space` can be found here: <https://travis-ci.org/haskell-perf/graphs>
 
 Results on bigger graphs and with more beautiful output can be found here: <https://github.com/haskell-perf/graphs/tree/master/results>
 
@@ -21,7 +22,7 @@ You can customize your build using several cabal flags (all of them are on by de
 
 * Time: will produce a benchmark suite using `criterion`.
 * Space: will produce a benchmark suite using `weigh`.
-* Datasize: will produce a benchmark suite using `ghc-datasize`.
+* Datasize: will produce a benchmark suite using `ghc-datasize` (disabled by default).
 
 ### Libraries
 
@@ -63,9 +64,13 @@ The benchmark suite `datasize` will use `ghc-datasize` to calculate size of grap
 ### Arguments
 
 Command-line arguments are self-explaining, but the `--graph "(String,Int)"` requires some explanations:
-We test functions against standards graphs, and they are built with ten-powers vertices. For standard graphs, the Int supplied is the upper-bound of the ten-powers. So `"(Path,100)"` will generate the `Path` with `1`, `10` and `100` vertices. You can specify several graphs.
+Standards graphs are built with ten-powers vertices. The `Int` supplied is the upper-bound of these ten-powers. So `"(Path,3)"` will generate three `Path` with `1`, `10` and `100` vertices. You can specify several graphs.
+
+You can force the suite to use only bigger graphs with the `-i` flag.
+
+The default is: `[("Mesh",3),("Clique",3)]`.
 
-The default is: `[("Mesh",3),("Clique",3)]`
+For real-life graphs (see below) you cannot use an integer greater than 4.
 
 #### Real-life graphs
 
@@ -86,6 +91,14 @@ The following graphs are supported:
 * Clique
 * RealLife (Note that because we have a limited set, you cannot request more than 4 real-life graphs)
 
+### Charts
+
+One can produce a chart from the results, use:
+
+```
+  -f,--chartfile FILENAME  Output file WITHOUT extension
+  -c,--chart OUTTYPE       Output type: Png or Svg
+```
 
 ## About implementation
 
@@ -131,7 +144,6 @@ data Suite g = forall i o. NFData o => Suite
 
 This module defines common builders for `Suite`, and particularly provides stable names for standard operations on graphs, and thus allows for simpler comparison (remember, benchmarks are identified by their _name_).
 
-
 ### Benchmarking with creation?
 
 We provide two ways of benchmarking:

bench/Space.hs

@@ -1,7 +1,7 @@
 {-# OPTIONS_GHC -fno-warn-incomplete-patterns #-}
 {-# LANGUAGE CPP #-}
 
-import Data.List (nub, nubBy, sortBy, elemIndices)
+import Data.List (nub, nubBy, sort, sortBy, elemIndices)
 import Data.Function (on)
 import Data.Maybe (mapMaybe, catMaybes, isJust)
 import Data.Int (Int64)
@@ -67,7 +67,7 @@ takeLastAfterBk w = case elemIndices '/' w of
 useResults :: Output -> [Named (Named String)] -> [Grouped (Weight, Maybe String)] -> IO ()
 useResults flg notDef todo = do
   putStrLn "Note: results are in bytes"
-  results <- mapM mapped $ nubBy (liftExtract2 eqG) namedBenchs
+  results <- fmap catMaybes $ mapM mapped $ nubBy (liftExtract2 eqG) namedBenchs
   maybe (return ()) (\x -> writeFile x $ show results) $ saveToFile flg
   case figOut flg of
     Nothing -> return ()
@@ -91,9 +91,9 @@ useResults flg notDef todo = do
                   printAbstract "lighter" onlyLargeBenchs
                 return $ Just (showGrouped $ snd e, onlyLargeBenchs)
 
-renderG :: T.ChartOutput -> [Maybe (Named (T.Grouped [Named Double]))] -> IO ()
+renderG :: T.ChartOutput -> [Named (T.Grouped [Named Double])] -> IO ()
 #ifdef CHART
-renderG x results = mkChart "Space results" defaultGr show x $ Left $ catMaybes results
+renderG x results = mkChart "Space results" defaultGr show x $ Left $ sortBy (on compare fst) results
 #else
 renderG _ _ = return ()
 #endif
@@ -189,7 +189,7 @@ main' (Run only notonly flg libs _ _ _) = do
 benchsNames :: Maybe Option -> Maybe [String] -> [String]
 benchsNames only notonly = nub $ useNotOnly $ useOnly extractedNames
   where
-    extractedNames = "creation" : map (\(_,Shadow s) -> either fst name s) listOfSuites
+    extractedNames = sort $ "creation" : map (\(_,Shadow s) -> either fst name s) listOfSuites
     useOnly = case only of
       Nothing -> id
       (Just (Only lst)) -> filter (`elem` lst)

bench/Time.hs

@@ -79,11 +79,11 @@ genReport gr flg arr = do
   unless notquickComp $ putStrLn $ let comp = head libNames
                                        oth =  head $ tail libNames
                                    in unwords ["\nComparing",comp,"to",oth,". It means that the displayed number will be k such that", comp,"= k *", oth ]
-  results <- mapM mapped $ nubBy (liftExtract2 (==)) refinedarr
+  results <- fmap catMaybes $ mapM mapped $ nubBy (liftExtract2 (==)) refinedarr
   maybe (return ()) (\x -> writeFile x $ unlines [show gr,show results]) $ saveToFile flg
   case figOut flg of
     Nothing -> return ()
-    (Just x) -> renderG gr x results
+    (Just x) -> renderG gr x  results
   where
     mapped e = do
       let bname = showBenchName $ snd e
@@ -110,9 +110,9 @@ genReport gr flg arr = do
     notquickComp = staOut flg /= QuickComparison
     (noimpl,refinedarr) = partitionEithers $ map stripOutEither arr
 
-renderG :: [(String,Int)] -> ChartOutput -> [Maybe (Name, Grouped [(Name, (Double, Double))])] -> IO ()
+renderG :: [(String,Int)] -> ChartOutput -> [(Name, Grouped [(Name, (Double, Double))])] -> IO ()
 #ifdef CHART
-renderG gr x results = mkChart "Time results" gr secs x $ Right $ catMaybes results
+renderG gr x results = mkChart "Time results" gr secs x $ Right $ sortBy (on compare fst) results
 #else
 renderG _ _ _ = return ()
 #endif
@@ -210,7 +210,7 @@ main' opts
                         Libs -> putStr $ unlines $ nub $ map fst listOfSuites ++ map fst (listOfCreation False [])
       Render filep dg -> do
         (gr,res) <- span (/='\n') <$> readFile filep
-        renderG (read gr) dg (read res)
+        renderG (read gr) dg  $ read res
       Run opt nottodo' flg libs benchWithCreation dontBenchLittleOnes gr' -> do
         let modifyL = case libs of
               Nothing -> id
@@ -231,7 +231,7 @@ main' opts
         genReport gr flg samples
   where
     grNames = nub $ map (either fst showBenchName . snd) $ grList False False defaultGr
-    grList benchWithCreation dontBenchLittleOnes gr =
+    grList benchWithCreation dontBenchLittleOnes gr = sortBy (on compare (either fst showBenchName . snd)) $
       map (fmap (\(Shadow s) -> second (allBench benchWithCreation dontBenchLittleOnes gr) s)) listOfSuites
       ++ listOfCreation dontBenchLittleOnes gr
     mkGr gr' = case gr' of

results/README.md

@@ -1,9 +1,12 @@
 ## Results
 
 * Time: https://github.com/haskell-perf/graphs/blob/master/results/TIME.md
+
+![svg](https://raw.githubusercontent.com/haskell-perf/graphs/newStack/results/TIME.svg?sanitize=true)
+
 * Space: https://github.com/haskell-perf/graphs/blob/master/results/SPACE.md
 
-Note: Some functions of Data.Graph was not in the original library (`edgeCount` and `hasEdge`), their defintion can be found in `bench/Containers/Graph.hs`
+Note: Some functions of some libraries are not implemented so we use a "hand-made" implementation. Defintions can be found in `bench/`. Suggestions are welcome!
 
 ## What is benchmarked
 
@@ -14,7 +17,7 @@ In this folder, you will find the lastest benchmarks of 4 haskell graphs librari
 * [FGL v5.6.0.0](https://hackage.haskell.org/package/fgl-5.6.0.0)
 * [Hash-Graph (not yet on hackage)](https://github.com/patrickdoc/hash-graph)
 
-The benchmarks were realised using `Stack` and the `stack.newest.yaml` configuration
+The benchmarks were realised using `Stack` and the `stack.yaml` configuration
 
 ## Tools
 
@@ -27,11 +30,22 @@ The benchmarks were realised using `Stack` and the `stack.newest.yaml` configura
 ### Benchmarking routine
 
 For the main results, we produce a generic list of edges (in ascending orders, but none of the libraries rely on this), create a graph from it, fully evaluate this graph to Normal Form, then pass it to then benchmarked function.
-This may not reflect the reality, so we produced https://github.com/haskell-perf/graphs/blob/master/results/TIME-creation.md where the creation time (from a list of edges) is taken into account.
+
+#### Creation
+
+This may not reflect the reality, so we produced an alternative table where creation time (from a list of edges) is taken into account:
+
+![svg](https://raw.githubusercontent.com/haskell-perf/graphs/newStack/results/TIME-creation.svg?sanitize=true)
+
+<https://github.com/haskell-perf/graphs/blob/master/results/TIME-creation.md> 
 
 ### The list of edges
 
-Containers, Fgl and Hash-Graph are dealing well with a list of edges. This is *not* the case with Alga, so we produced https://github.com/haskell-perf/graphs/blob/master/results/TIME-extra.md where we used the alga representation instead of a list of edges.
+Containers, Fgl and Hash-Graph are dealing well with a list of edges. This is *not* the case with Alga, so we produced an alternative table where we used the alga representation instead of a list of edges:
+
+![svg](https://raw.githubusercontent.com/haskell-perf/graphs/newStack/results/TIME-extra.svg?sanitize=true)
+
+<https://github.com/haskell-perf/graphs/blob/master/results/TIME-extra.md>
 
 ## Some words about graphs
 The functions are benchmarked against:
@@ -56,9 +70,11 @@ The functions are benchmarked against:
 The two first graphs are built with successive ten powers vertices. Here, with 1, 10, 100 and 1000 vertices.
 
 ### Types
+
 Libraries are benchmarked against graphs with *Int* vertices.
 
 ## About arguments
+
 All the functions are tested with arguments in the _domain_ of the graph, where applicable: unless it is mentioned, edges and vertices generated for the test can be in the complete graph with the same number of vertices.
 
 ## Remarks