feat: add bootstrap command and implement bootstrap functionality for Deckhouse modules #233

juev · 2025-07-09T14:23:50Z

This pull request introduces a new bootstrap command for Deckhouse modules, refactors the existing gen command, and improves error handling and test validation in various parts of the codebase. The most significant changes include adding the bootstrap command functionality, updating flag definitions, enhancing test assertions, and improving HTTP client behavior.

PR Description: Add Bootstrap Command for Deckhouse Module Development

🎯 Overview

This PR introduces a new bootstrap command to the Deckhouse Module Tool (dmt), enabling developers to quickly scaffold new Deckhouse modules from an official template. This significantly streamlines the module development workflow by automating the initial setup process.

🚀 Key Features

New Bootstrap Command

Purpose: Initialize a new Deckhouse module from template

Multi-Platform CI/CD Support

GitHub Integration: Automatically configures GitHub Actions workflows
GitLab Integration: Sets up GitLab CI/CD pipelines

Template Management

Default Template: Uses official Deckhouse modules template from GitHub
Custom Templates: Supports custom template repositories via URL

📋 Use Cases

1. Quick Module Development Setup

# Create a new GitHub-based module
dmt bootstrap payment-gateway

# Create a GitLab-based module  
dmt bootstrap user-management --pipeline gitlab

# Bootstrap in specific directory
dmt bootstrap monitoring-extension --directory ./modules/monitoring

2. Enterprise/Custom Templates

# Use organization-specific template
dmt bootstrap internal-tool --repository-url https://github.com/myorg/custom-template/archive/main.zip

3. Team Standardization

Ensures consistent module structure across team
Automates proper naming conventions
Pre-configures CI/CD based on organization preferences

🔧 Technical Implementation

Command Structure

Input Validation: Module name must be in kebab-case format
Platform Validation: Repository type must be github or gitlab
Directory Safety: Ensures target directory is empty before proceeding

Template Processing

Download: Fetches template ZIP from repository URL
Extract: Safely extracts content with size limits
Transform: Replaces template placeholders with actual module name
Configure: Sets up appropriate CI/CD files for chosen platform
Cleanup: Removes unnecessary files based on platform choice

Smart String Replacement

Module Names: Replaces all template module references
Values References: Converts .Values.templateName to .Values.moduleName (camelCase)
Internal References: Updates templateName.internal patterns

🎛️ Configuration Options

Flag	Short	Default	Description
`--pipeline`	`-p`	`github`	CI/CD platform (`github` or `gitlab`)
`--directory`	`-d`	`.`	Target directory for bootstrap
`--repository-url`	`-r`		Custom template repository URL

📖 Post-Bootstrap Guidance

The command provides platform-specific setup instructions.

This bootstrap functionality dramatically reduces the time and effort required to start developing new Deckhouse modules, ensuring consistency and best practices across all module projects.

… Deckhouse modules

…r improved flexibility - Modified `downloadAndExtractTemplate` and `moveExtractedContent` functions to accept a directory parameter. - Updated tests to reflect changes in function signatures and ensure correct directory handling during file operations.

- Updated `InitBootstrapFlagSet` to use shorthand flags for repository type, URL, and directory parameters, improving usability and flexibility.

…ction signature - Added validation to ensure a module name is provided when executing the bootstrap command. - Updated `RunBootstrap` function to accept the module name as a parameter. - Implemented `getModuleName` function to read and extract the module name from `module.yaml` file.

…unctionality - Added tests for `RunBootstrap`, `replaceModuleName`, and `replaceValuesModuleName` to ensure correct functionality. - Implemented logic to replace module names in files and update references in YAML configurations. - Enhanced error handling in tests for invalid scenarios, including file path and URL checks.

…ctionality - Renamed package from `bootstap` to `bootstrap` for consistency and clarity. - Updated `bootstrapCmdFunc` to utilize a structured `BootstrapConfig` for improved readability and maintainability. - Added comprehensive tests for `RunBootstrap` and related functions to ensure robust functionality and error handling.

- Updated the WriteProto method to use http.NewRequestWithContext for better context propagation. - Ensured that the HTTP request utilizes the provided context for improved request handling and cancellation support.

- Replaced assertions with require statements in bootstrap tests for better error handling and test clarity. - Updated file permission constants for consistency and security. - Enhanced the downloadFile function to limit file size, preventing potential DoS attacks. - Cleaned up test code by removing unused helper functions and ensuring proper directory restoration in tests.

- Added error logging for bootstrap command execution failures to enhance observability. - Implemented validation for repository types in RunBootstrap to ensure only valid types are processed. - Added directory creation logic if the specified directory does not exist, improving robustness. - Enhanced cleanup logic to remove unnecessary directories and files based on repository type.

- Modified downloadAndExtractTemplate function to accept BootstrapConfig instead of a directory string for improved flexibility. - Updated repository URL handling to utilize the config's RepositoryURL if provided. - Adjusted calls to moveExtractedContent to use the directory from the BootstrapConfig, enhancing code clarity and maintainability.

- Added tests for various bootstrap scenarios including GitLab repository type, invalid repository types, and non-existent directories. - Improved error handling tests for file operations, ensuring robustness against read/write errors and invalid paths. - Enhanced test coverage for the RunBootstrap function, validating expected outcomes and error messages.

- Introduced the bootstrap package with core functionalities for initializing modules. - Added tests for various scenarios including successful bootstraps, handling non-empty directories, and validating module name replacements. - Enhanced error handling in tests to cover edge cases such as invalid repository types and file operations. - Implemented logic for downloading and extracting templates, ensuring robust directory management and file operations.

- Added tests for the downloadFile function using a mock HTTP server to simulate various scenarios, including successful downloads, invalid URLs, and server errors. - Improved error assertions to ensure proper handling of download failures and file creation issues. - Updated existing tests to utilize the mock server for consistency and reliability in testing file downloads.

- Enhanced the TestDownloadFile function to include error handling for zip file creation and writing operations. - Updated assertions to use assert.Positive for better clarity in verifying downloaded file size. - Improved error handling in mock server responses to ensure robustness against potential failures during file operations.

Copilot

Pull Request Overview

This PR introduces a new bootstrap command to initialize Deckhouse modules from a template, including flag support, the core bootstrap logic, and comprehensive tests.

Add bootstrap subcommand with flags for repository type, URL, and target directory
Implement bootstrap workflow in internal/bootstrap (download, extract, replace names, cleanup)
Integrate command in cmd/dmt/root.go and update flag definitions; minor HTTP client context improvement in promremote.go

Reviewed Changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
internal/promremote/promremote.go	Use `NewRequestWithContext` instead of `Do(req.WithContext)`
internal/flags/flags.go	Add `Bootstrap*` globals and `InitBootstrapFlagSet`
internal/bootstrap/bootstrap.go	Implement full bootstrap logic
internal/bootstrap/bootstrap_test.go	Add tests covering bootstrap behavior and errors
cmd/dmt/root.go	Register `bootstrap` command and handler

Comments suppressed due to low confidence (2)

cmd/dmt/root.go:65

[nitpick] The Long description is very generic; consider updating it to describe the command's purpose (e.g., "Initializes a new Deckhouse module from a template").

		Long:  `A lot of useful bootstraps`,

internal/flags/flags.go:70

[nitpick] Consider enhancing this flag's help text to clarify the allowed values and default behavior; for example, note that the default repository type is GitHub.

	bootstrap.StringVarP(&BootstrapRepositoryType, "repository-type", "t", "github", "repository type [github | gitlab]")

internal/bootstrap/bootstrap.go

…tion - Skipped integration tests that require a real template archive with module.yaml for the TestRunBootstrap and TestRunBootstrapWithNonExistentDirectory functions. - Enhanced TestExtractZip to verify direct extraction of files without a root directory and added checks for module.yaml. - Adjusted error assertions in TestExtractZipNoRootDirectory to reflect changes in expected error messages. - Improved directory creation logic in extractZip to use consistent permissions.

- Added directory creation logic in TestExtractZip to ensure the extraction directory exists before extraction. - Enhanced assertions to verify the structure and content of extracted files, including checks for root directory and file contents. - Updated error assertions in TestExtractZipNoRootDirectory to reflect the correct expected error message. - Improved logging for extracted files to aid in debugging and verification of extraction results.

- Included reminders in the RunBootstrap function to inform users about necessary environment variables for CI/CD pipelines. - Enhanced user experience by providing clear instructions on required variables for external modules and registry configurations.

…validation - Added BootstrapModule flag to specify the module name during bootstrap. - Implemented validation for BootstrapRepositoryType to ensure it is either 'github' or 'gitlab'. - Updated bootstrap command logic to handle errors for missing module name and invalid repository type.

- Refactored the moveExtractedContent function to utilize moveFileOrDirectory for moving files and directories. - Implemented moveFileOrDirectory with a fallback to copy-and-remove for cross-filesystem compatibility. - Added copyAndRemove, copyDirectoryAndRemove, and copyFileAndRemove functions to handle file and directory copying and removal. - Enhanced error handling and logging for file operations to improve robustness and traceability.

…ory type - Updated the RunBootstrap function to display reminders for necessary CI/CD variables only when the repository type is GitLab. - Enhanced user guidance by ensuring relevant instructions are provided based on the selected repository configuration.

…rap tests - Deleted TestRunBootstrapWithInvalidRepositoryType as it was deemed unnecessary. - Focused on maintaining relevant test cases that align with current repository type validation logic.

- Updated error assertions in crds_test.go to create a map of expected error messages for easier comparison. - Implemented checks to ensure that all actual errors match expected errors, regardless of order. - Improved test robustness by ensuring unexpected errors are clearly reported.

…bootstrap Signed-off-by: Pavel Okhlopkov <[email protected]>

Signed-off-by: Pavel Okhlopkov <[email protected]>

…dation and run Signed-off-by: Pavel Okhlopkov <[email protected]>

Signed-off-by: Pavel Okhlopkov <[email protected]>

…ions Signed-off-by: Pavel Okhlopkov <[email protected]>

Signed-off-by: Pavel Okhlopkov <[email protected]>

feat: add bootstrap command and implement bootstrap functionality for…

b91ebdb

… Deckhouse modules

juev self-assigned this Jul 9, 2025

Evsyukov Denis added 12 commits July 10, 2025 11:59

refactor: enhance bootstrap flag definitions with shorthand options

02541c9

- Updated `InitBootstrapFlagSet` to use shorthand flags for repository type, URL, and directory parameters, improving usability and flexibility.

fix: add context to HTTP request in WriteProto method

86e26b5

- Updated the WriteProto method to use http.NewRequestWithContext for better context propagation. - Ensured that the HTTP request utilizes the provided context for improved request handling and cancellation support.

chore: update test coverage badge [skip ci]

07ce845

juev requested a review from Copilot July 10, 2025 12:08

This comment was marked as outdated.

Sign in to view

Evsyukov Denis added 2 commits July 10, 2025 15:13

juev requested a review from Copilot July 10, 2025 12:22

Copilot AI reviewed Jul 10, 2025

View reviewed changes

internal/bootstrap/bootstrap.go Show resolved Hide resolved

internal/bootstrap/bootstrap.go Outdated Show resolved Hide resolved

Evsyukov Denis added 2 commits July 10, 2025 15:38

juev force-pushed the feature/bootstrap branch from 66dcdd2 to b979586 Compare July 10, 2025 13:41

Evsyukov Denis added 7 commits July 10, 2025 13:45

chore: update test coverage badge [skip ci]

e42e2e0

chore: update test coverage badge [skip ci]

14c460a

refactor: remove redundant test for invalid repository type in bootst…

54a4ac1

…rap tests - Deleted TestRunBootstrapWithInvalidRepositoryType as it was deemed unnecessary. - Focused on maintaining relevant test cases that align with current repository type validation logic.

Evsyukov Denis added 2 commits July 11, 2025 11:01

chore: update test coverage badge [skip ci]

96b33c1

juev requested a review from ldmonster July 11, 2025 09:36

juev marked this pull request as ready for review July 11, 2025 09:37

Evsyukov Denis and others added 9 commits July 14, 2025 13:24

feat: add conditional reminders for GitHub and GitLab repository types

5c1a050

chore: update test coverage badge [skip ci]

f7065e7

fix: update reminder message for GitLab CI/CD variables

a0cb821

Merge commit '1cfb03982dd29ed4b6b7fe5ffc04a753fc21b856' into feature/…

dae7f29

…bootstrap Signed-off-by: Pavel Okhlopkov <[email protected]>

remove empty run which blocked help

427d12c

Signed-off-by: Pavel Okhlopkov <[email protected]>

add module name as required arg, add default dir, split logic to vali…

2868847

…dation and run Signed-off-by: Pavel Okhlopkov <[email protected]>

bump readme

ef4be49

Signed-off-by: Pavel Okhlopkov <[email protected]>

remove useless linter. we know when we use shadowing in err consctuct…

15e9ade

…ions Signed-off-by: Pavel Okhlopkov <[email protected]>

lint

07586ab

Signed-off-by: Pavel Okhlopkov <[email protected]>

ldmonster added the enhancement New feature or request label Aug 12, 2025

ldmonster added 2 commits August 12, 2025 15:38

add color, add debug logs

a8f0246

Signed-off-by: Pavel Okhlopkov <[email protected]>

replace config val with calculated one

6afcbeb

Signed-off-by: Pavel Okhlopkov <[email protected]>

ldmonster merged commit acd3dc1 into main Aug 12, 2025
3 checks passed

ldmonster deleted the feature/bootstrap branch August 12, 2025 13:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add bootstrap command and implement bootstrap functionality for Deckhouse modules #233

feat: add bootstrap command and implement bootstrap functionality for Deckhouse modules #233

Uh oh!

juev commented Jul 9, 2025 •

edited by ldmonster

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

feat: add bootstrap command and implement bootstrap functionality for Deckhouse modules #233

feat: add bootstrap command and implement bootstrap functionality for Deckhouse modules #233

Uh oh!

Conversation

juev commented Jul 9, 2025 • edited by ldmonster Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Description: Add Bootstrap Command for Deckhouse Module Development

🎯 Overview

🚀 Key Features

New Bootstrap Command

Multi-Platform CI/CD Support

Template Management

📋 Use Cases

1. Quick Module Development Setup

2. Enterprise/Custom Templates

3. Team Standardization

🔧 Technical Implementation

Command Structure

Template Processing

Smart String Replacement

🎛️ Configuration Options

📖 Post-Bootstrap Guidance

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juev commented Jul 9, 2025 •

edited by ldmonster

Loading