Skip to content

Commit e9ce453

Browse files
jphuynhjphuynh
authored
feat: Optionally Limit the number of concurrent go routines via SizedWaitGroup (#38)
1 parent d2d6079 commit e9ce453

File tree

8 files changed

+32
-14
lines changed

8 files changed

+32
-14
lines changed

‎README.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -324,16 +324,17 @@ echo "gruntwork-io/terragrunt gruntwork-io/terratest" | git-xargs \
324324
325325
`git-xargs` exposes several flags that allow you to customize its behavior to better suit your needs. For the latest info on flags, you should run `git-xargs --help`. However, a couple of the flags are worth explaining more in depth here:
326326
327-
| Flag | Description | Type | Required |
328-
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- |
329-
| `--branch-name` | You must specify the name of the branch to make your local and remote changes on. You can further control branching behavior via `--skip-pull-requests` as explained below | String | Yes |
330-
| `--repos` | If you want to specify many repos and manage them in files (which makes batching and testing easier) then use this flag to pass the filepath to a repos file. See [the repos file format](#option-2-flat-file-of-repository-names) for more information | String | No |
331-
| `--repo` | Use this flag to specify a single repo, e.g., `--repo gruntwork-io/cloud-nuke`. Can be passed multiple times to target several repos | String | No |
332-
| `--github-org` | If you want to target every repo in a Github org that your GITHUB_OAUTH_TOKEN has access to, pass the name of the Organization with this flag, to page through every repo via the Github API and target it | String | No |
333-
| `--commit-message` | The commit message to use when creating commits. If you supply this flag, but neither the optional `--pull-request-title` or `--pull-request-description` flags, then the commit message value will be used for all three. | String | No |
334-
| `--skip-pull-requests` | If you don't want any pull requests opened, but would rather have your changes committed directly to your specified branch, pass this flag. Note that it won't work if your Github repo is configured with branch protections on the branch you're trying to commit directly to! | Boolean | No |
335-
| `--skip-archived-repos`| If you want to exclude archived (read-only) repositories from the list of targeted repos, pass this flag. | Boolean | No |
336-
| `--dry-run` | If you are in the process of testing out `git-xargs` or your intial set of targeted repos, but you don't want to make any changes via the Github API (pushing your local changes or opening pull requests) you can pass the dry-run branch. This is useful because the output report will still tell you which repos would have been affected, without actually making changes via the Github API to your remote repositories. | Boolean | No |
327+
| Flag | Description | Type | Required |
328+
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | -------- |
329+
| `--branch-name` | You must specify the name of the branch to make your local and remote changes on. You can further control branching behavior via `--skip-pull-requests` as explained below | String | Yes |
330+
| `--repos` | If you want to specify many repos and manage them in files (which makes batching and testing easier) then use this flag to pass the filepath to a repos file. See [the repos file format](#option-2-flat-file-of-repository-names) for more information | String | No |
331+
| `--repo` | Use this flag to specify a single repo, e.g., `--repo gruntwork-io/cloud-nuke`. Can be passed multiple times to target several repos | String | No |
332+
| `--github-org` | If you want to target every repo in a Github org that your GITHUB_OAUTH_TOKEN has access to, pass the name of the Organization with this flag, to page through every repo via the Github API and target it | String | No |
333+
| `--commit-message` | The commit message to use when creating commits. If you supply this flag, but neither the optional `--pull-request-title` or `--pull-request-description` flags, then the commit message value will be used for all three. | String | No |
334+
| `--skip-pull-requests` | If you don't want any pull requests opened, but would rather have your changes committed directly to your specified branch, pass this flag. Note that it won't work if your Github repo is configured with branch protections on the branch you're trying to commit directly to! | Boolean | No |
335+
| `--skip-archived-repos` | If you want to exclude archived (read-only) repositories from the list of targeted repos, pass this flag. | Boolean | No |
336+
| `--dry-run` | If you are in the process of testing out `git-xargs` or your intial set of targeted repos, but you don't want to make any changes via the Github API (pushing your local changes or opening pull requests) you can pass the dry-run branch. This is useful because the output report will still tell you which repos would have been affected, without actually making changes via the Github API to your remote repositories. | Boolean | No |
337+
| `--max-concurrent-repos` | Limits the number of concurrent processed repositories. This is only useful if you encounter issues and need throttling when running on a very large number of repos. Default is `0` (Unlimited) | Integer | No |
337338
338339
## Best practices, tips and tricks
339340

‎cmd/git-xargs.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ func parseGitXargsConfig(c *cli.Context) (*config.GitXargsConfig, error) {
3030
config.ReposFile = c.String("repos")
3131
config.GithubOrg = c.String("github-org")
3232
config.RepoSlice = c.StringSlice("repo")
33+
config.MaxConcurrentRepos = c.Int("max-concurrent-repos")
3334
config.Args = c.Args()
3435

3536
shouldReadStdIn, err := dataBeingPipedToStdIn()

‎common/common.go

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,11 @@ const (
1313
BranchFlagName = "branch-name"
1414
PullRequestTitleFlagName = "pull-request-title"
1515
PullRequestDescriptionFlagName = "pull-request-description"
16+
MaxConcurrentReposFlagName = "max-concurrent-repos"
1617
DefaultCommitMessage = "git-xargs programmatic commit"
1718
DefaultPullRequestTitle = "git-xargs programmatic pull request"
1819
DefaultPullRequestDescription = "git-xargs programmatic pull request"
20+
DefaultMaxConcurrentRepos = 0
1921
)
2022

2123
var (
@@ -62,4 +64,9 @@ var (
6264
Usage: "The description to add to pull requests opened by git-xargs",
6365
Value: DefaultPullRequestDescription,
6466
}
67+
GenericMaxConcurrentReposFlag = cli.IntFlag{
68+
Name: MaxConcurrentReposFlagName,
69+
Usage: "Limits the number of concurrent processed repositories. This is only useful if you encounter issues and need throttling when running on a very large number of repos. Default is 0 (Unlimited)",
70+
Value: DefaultMaxConcurrentRepos,
71+
}
6572
)

‎config/config.go

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ type GitXargsConfig struct {
1515
DryRun bool
1616
SkipPullRequests bool
1717
SkipArchivedRepos bool
18+
MaxConcurrentRepos int
1819
BranchName string
1920
CommitMessage string
2021
PullRequestTitle string
@@ -35,6 +36,7 @@ func NewGitXargsConfig() *GitXargsConfig {
3536
DryRun: false,
3637
SkipPullRequests: false,
3738
SkipArchivedRepos: false,
39+
MaxConcurrentRepos: 0,
3840
BranchName: "",
3941
CommitMessage: common.DefaultCommitMessage,
4042
PullRequestTitle: common.DefaultPullRequestTitle,

‎go.mod

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ require (
1010
github.com/kataras/tablewriter v0.0.0-20180708051242-e063d29b7c23
1111
github.com/landoop/tableprinter v0.0.0-20200805134727-ea32388e35c1
1212
github.com/mattn/go-runewidth v0.0.9 // indirect
13+
github.com/remeh/sizedwaitgroup v1.0.0
1314
github.com/sirupsen/logrus v1.7.0
1415
github.com/stretchr/testify v1.7.0
1516
github.com/urfave/cli v1.22.5

‎go.sum

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -430,6 +430,8 @@ github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7z
430430
github.com/prometheus/procfs v0.0.2/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
431431
github.com/prometheus/procfs v0.0.3/go.mod h1:4A/X28fw3Fc593LaREMrKMqOKvUAntwMDaekg4FpcdQ=
432432
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
433+
github.com/remeh/sizedwaitgroup v1.0.0 h1:VNGGFwNo/R5+MJBf6yrsr110p0m4/OX4S3DCy7Kyl5E=
434+
github.com/remeh/sizedwaitgroup v1.0.0/go.mod h1:3j2R4OIe/SeS6YDhICBy22RWjJC5eNCJ1V+9+NVNYlo=
433435
github.com/remyoudompheng/bigfft v0.0.0-20170806203942-52369c62f446/go.mod h1:uYEyJGbgTkfkS4+E/PavXkNJcbFIpEtjt2B0KDQ5+9M=
434436
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
435437
github.com/rogpeppe/go-charset v0.0.0-20180617210344-2471d30d28b4/go.mod h1:qgYeAmZ5ZIpBWTGllZSQnw97Dj+woV0toclVaRGI8pc=

‎main.go

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,7 @@ func setupApp() *cli.App {
7171
common.GenericCommitMessageFlag,
7272
common.GenericPullRequestTitleFlag,
7373
common.GenericPullRequestDescriptionFlag,
74+
common.GenericMaxConcurrentReposFlag,
7475
}
7576

7677
app.Action = cmd.RunGitXargs

‎repository/process.go

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,23 @@
11
package repository
22

33
import (
4-
"sync"
5-
64
"github.com/google/go-github/v32/github"
75
"github.com/gruntwork-io/git-xargs/config"
86
"github.com/gruntwork-io/go-commons/logging"
7+
"github.com/remeh/sizedwaitgroup"
98
"github.com/sirupsen/logrus"
109
)
1110

1211
// Loop through every repo we've selected and use a WaitGroup so that the processing can happen in parallel
1312
func ProcessRepos(gitxargsConfig *config.GitXargsConfig, repos []*github.Repository) error {
1413
logger := logging.GetLogger("git-xargs")
15-
var wg sync.WaitGroup
14+
15+
// Limit the number of concurrent goroutines using the MaxConcurrentRepos config value
16+
// MaxConcurrentRepos == 0 will fallback to unlimited (previous default behavior)
17+
wg := sizedwaitgroup.New(gitxargsConfig.MaxConcurrentRepos)
18+
1619
for _, repo := range repos {
17-
wg.Add(1)
20+
wg.Add()
1821
go func(gitxargsConfig *config.GitXargsConfig, repo *github.Repository) error {
1922
defer wg.Done()
2023
// For each repo, run the supplied command against it and, if it succeeds without error,

0 commit comments

Comments
 (0)