various updated prior to tune 2.0 (#1092)

* update urls

* doMC -> future

* doMC -> future

* updated example objects

* remove unused functions (and their direct tests)

* A note about require in the extract files

* polish news

* added cran skips for long running examples

* Apply suggestions from code review

Co-authored-by: Emil Hvitfeldt <[email protected]>

* updates based on reviewer comments

* not used

* pad zeros in iterative search .configs

* a few more foreach references

* run inst/test_objects.R

* reorder column names

* remove Rout file

* air format

* Apply suggestions from code review

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

---------

Co-authored-by: Emil Hvitfeldt <[email protected]>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Author: 3 people

NEWS.md

@@ -2,29 +2,37 @@
 
 ## Changes to `tune_grid()`. 
 
-* A major rewrite and refactor of the underlying code that runs `tune_grid()` was made. 
+* A major rewrite/refactor of the underlying code that runs `tune_grid()`. This was an upgrade to add postprocessing and to modernize our parallel processing infrastructure. 
 
-* The pattern of `.config` values has changed from `Preprocessor{num}_Model{num}` to `pre{num}_mod{num}_post{num}`. 
+* The pattern of `.config` values has changed. 
+
+   - For grid search, it changes from `Preprocessor{num}_Model{num}` to `pre{num}_mod{num}_post{num}`. The numbers include a zero when that element was static. For example, a value of `pre0_mod3_post4` means no preprocessors were tuned and the model and postprocessor(s) had at least three and four candidates, respectively. 
+   - For iterative search, the pattern is not `iter{num}` instead of `Iter{num}` and the numbers are now zero padded to sort better. For example, if there between 10 and 99 iterations, the first `.config` value is now `iter01` instead of `Iter1`.
 
 * The package will now log a backtrace for errors and warnings that occur during tuning. When a tuning process encounters issues, see the new `trace` column in the `collect_notes(.Last.tune.result)` output to find precisely where the error occurred (#873).
 
-* Post-processing: new `schedule_grid()` for scheduling a grid including post-processing (#988).
+* Postprocessors can now be tuned. Currently, we support the tailor package. 
 
-## Other Changes
+## Parallel Processing
 
 * Introduced support for parallel processing with mirai in addition to the currently supported framework future. See `?parallelism` to learn more (#1028).
 
 * Sequential and parallel processing all use the same L'Ecuyer-CMRG seeds (conditional on `parallel_over`) (#1033). 
 
-* `int_pctl()` now includes an option (`keep_replicates`) to retain the individual bootstrap estimates. It also processes the resamples more efficiently (#1000).
-
-* A `min_grid()` methods was added for `proportional_hazards` models so that their submodels are processed appropriately. 
-
 ## Breaking Changes
 
 * The `foreach` package is no longer supported. Instead, use the future or mirai packages.  
+
 * The parallel backend(s) and the methods of constructing seeds for workers have changed. There will be a lack of reproducibility between objects created in this version of tune and previous versions.
 
+## Other Changes
+
+* `int_pctl()` now includes an option (`keep_replicates`) to retain the individual bootstrap estimates. It also processes the resamples more efficiently (#1000).
+
+* A `min_grid()` methods was added for `proportional_hazards` models so that their submodels are processed appropriately. 
+
+* Post-processing: new `schedule_grid()` for scheduling a grid including post-processing (#988).
+
 # tune 1.3.0
 
 * The package will now warn when parallel processing has been enabled with foreach but not with future. See [`?parallelism`](https://tune.tidymodels.org/dev/reference/parallelism.html) to learn more about transitioning your code to future (#878, #866). The next version of tune will move to a pure future implementation. 

R/compute_metrics.R

@@ -147,9 +147,9 @@ compute_metrics.tune_results <- function(
   # nest by resample id
   nest_cols <- "id"
 
-  if ("Iter1" %in% mtrcs$.config) {
+  # Convert the iterative .configs into numbers
+  if (any(grepl("^[iI]ter", mtrcs$.config))) {
     mtrcs$.iter <- .config_to_.iter(mtrcs$.config)
-
     nest_cols <- c(nest_cols, ".iter")
   }
 

R/fit_best.R

@@ -37,7 +37,7 @@
 #'
 #' @inheritSection last_fit See also
 #'
-#' @examplesIf tune:::should_run_examples() & rlang::is_installed("modeldata")
+#' @examplesIf tune:::should_run_examples() && rlang::is_installed("modeldata") && !tune:::is_cran_check()
 #' library(recipes)
 #' library(rsample)
 #' library(parsnip)

R/grid_helpers.R

@@ -99,15 +99,6 @@ forge_from_workflow <- function(new_data, workflow) {
   forged
 }
 
-mold_has_case_weights <- function(mold) {
-  roles <- mold$extras$roles
-  no_extras <- is.null(roles)
-  if (no_extras) {
-    return(FALSE)
-  }
-  any(names(roles) == "case_weights")
-}
-
 get_metrics_by <- function(metric_set) {
   metrics <- attr(metric_set, "metrics")
   metrics_by <- purrr::map(metrics, attr, "by")

R/loop_over_all_stages-helpers.R

@@ -370,21 +370,6 @@ replace_reserve_rows <- function(iter, chunk) {
   start_loc:end_loc
 }
 
-update_reserve <- function(reserve, iter, predictions, grid_size) {
-  grid_size <- min(1, grid_size)
-  pred_size <- nrow(predictions)
-
-  if (is.null(reserve)) {
-    reserve <- initialize_pred_reserve(predictions, grid_size)
-  } else {
-    if (tibble::is_tibble(predictions)) {
-      predictions <- dplyr::as_tibble(predictions)
-    }
-  }
-  reserve[replace_reserve_rows(iter, pred_size), ] <- predictions
-  reserve
-}
-
 # ------------------------------------------------------------------------------
 # Add .config to grid
 

R/parallel.R

@@ -483,16 +483,6 @@ pctl_call <- function(framework, args = list()) {
 
 # ------------------------------------------------------------------------------
 
-warn_foreach_deprecation <- function() {
-  cli::cli_warn(c(
-    "!" = "{.pkg tune} detected a parallel backend registered with \\
-               foreach but no backend registered with future.",
-    "i" = "Support for parallel processing with foreach was \\
-               soft-deprecated in {.pkg tune} 1.2.1.",
-    "i" = "See {.help [?parallelism](tune::parallelism)} to learn more."
-  ))
-}
-
 manange_global_limit <- function(min = 1e9) {
   currrent_value <- getOption("future.globals.maxSize")
   if (is.null(currrent_value)) {

R/pull.R

@@ -5,79 +5,6 @@ extract_details <- function(object, extractor) {
   extractor(object)
 }
 
-# ------------------------------------------------------------------------------
-
-# Grab the new results, make sure that they align row-wise with the rsample
-# object and then bind columns
-pulley <- function(resamples, res, col, order) {
-  if (all(purrr::map_lgl(res, inherits, "simpleError"))) {
-    res <-
-      resamples |>
-      mutate(col = purrr::map(splits, \(x) NULL)) |>
-      setNames(c(names(resamples), col))
-    return(res)
-  }
-
-  all_null <- all(purrr::map_lgl(res, is.null))
-
-  id_cols <- grep("^id", names(resamples), value = TRUE)
-
-  resamples <- vctrs::vec_slice(resamples, order)
-
-  pulled_vals <- purrr::map(res, \(.x) .x[[col]]) |> purrr::list_rbind()
-
-  if (nrow(pulled_vals) == 0) {
-    res <-
-      resamples |>
-      mutate(col = purrr::map(splits, \(x) NULL)) |>
-      setNames(c(names(resamples), col))
-    return(res)
-  }
-
-  pulled_vals <- tidyr::nest(pulled_vals, data = -starts_with("id"))
-  names(pulled_vals)[ncol(pulled_vals)] <- col
-
-  res <- new_bare_tibble(resamples)
-  res <- full_join(res, pulled_vals, by = id_cols)
-  res <- reup_rs(resamples, res)
-  res
-}
-
-maybe_repair <- function(x) {
-  not_null <- !purrr::map_lgl(x, is.null)
-  is_tibb <- purrr::map_lgl(x, tibble::is_tibble)
-  ok <- not_null & is_tibb
-  if (!any(ok)) {
-    return(x)
-  }
-
-  good_val <- which(ok)[1]
-  template <- x[[good_val]][0, ]
-
-  insert_val <- function(x, y) {
-    if (is.null(x)) {
-      x <- y
-    }
-    x
-  }
-
-  x <- purrr::map(x, insert_val, y = template)
-  x
-}
-
-ensure_tibble <- function(x) {
-  if (is.null(x)) {
-    res <- tibble::new_tibble(list(.notes = character(0)), nrow = 0)
-  } else {
-    res <- tibble::new_tibble(list(.notes = x), nrow = length(x))
-  }
-  res
-}
-
-append_outcome_names <- function(all_outcome_names, outcome_names) {
-  c(all_outcome_names, list(outcome_names))
-}
-
 #' Convenience functions to extract model
 #'
 #' `r lifecycle::badge("soft-deprecated")`

R/tune_bayes.R

@@ -61,6 +61,15 @@
 #' results. For good results, the number of initial values should be more than
 #' the number of parameters being optimized.
 #'
+#' The tuning parameter combinations that were tested are called _candidates_.
+#'  Each candidate has a unique `.config` value that, for the initial grid search,
+#'  has the pattern `pre{num}_mod{num}_post{num}`. The numbers include a zero
+#'  when that element was static. For example, a value of `pre0_mod3_post4` means
+#'  no preprocessors were tuned and the model and postprocessor(s) had at least
+#'  three and four candidates, respectively. The iterative part of the
+#'  search uses the pattern `iter{num}`. In each case, the numbers are
+#'  zero-padded to enable proper sorting.
+#'
 #' @section Parameter Ranges and Values:
 #'
 #' In some cases, the tuning parameter values depend on the dimensions of the
@@ -135,7 +144,7 @@
 #' calculated for every value of `eval_time` but the _first_ evaluation time
 #' given by the user (e.g., `eval_time[1]`) is used to guide the optimization.
 #'
-#' @examplesIf tune:::should_run_examples(suggests = "kernlab")
+#' @examplesIf tune:::should_run_examples(suggests = "kernlab") && !tune:::is_cran_check()
 #' library(recipes)
 #' library(rsample)
 #' library(parsnip)
@@ -290,6 +299,11 @@ tune_bayes_workflow <- function(
   rset_info <- pull_rset_attributes(resamples)
 
   check_iter(iter, call = call)
+  if (iter > 0) {
+    iter_chr <- recipes::names0(iter, "iter")
+  } else {
+    iter_chr <- "iter0"
+  }
 
   metrics <- check_metrics_arg(metrics, object, call = call)
   opt_metric <- first_metric(metrics)
@@ -488,20 +502,20 @@ tune_bayes_workflow <- function(
         tmp_res[[".metrics"]] <- purrr::map(
           tmp_res[[".metrics"]],
           dplyr::mutate,
-          .config = paste0("Iter", i)
+          .config = iter_chr[i]
         )
         if (control$save_pred) {
           tmp_res[[".predictions"]] <- purrr::map(
             tmp_res[[".predictions"]],
             dplyr::mutate,
-            .config = paste0("Iter", i)
+            .config = iter_chr[i]
           )
         }
         if (".extracts" %in% names(tmp_res)) {
           tmp_res[[".extracts"]] <- purrr::map(
             tmp_res[[".extracts"]],
             dplyr::mutate,
-            .config = paste0("Iter", i)
+            .config = iter_chr[i]
           )
         }
         unsummarized <- dplyr::bind_rows(

R/tune_grid.R

@@ -77,6 +77,14 @@
 #'  change the values. This updated parameter set can be passed to the function
 #'  via the `param_info` argument.
 #'
+#'  The rows of the grid are called tuning parameter _candidates_. Each
+#'  candidate has a unique `.config` value that, for grid search, has the
+#'  pattern `pre{num}_mod{num}_post{num}`. The numbers include a zero when that
+#'  element was static. For example, a value of `pre0_mod3_post4` means no
+#'  preprocessors were tuned and the model and postprocessor(s) had at least
+#'  three and four candidates, respectively. Also, the numbers are zero-padded
+#'  to enable proper sorting.
+#'
 #' @section Performance Metrics:
 #'
 #' To use your own performance metrics, the [yardstick::metric_set()] function
@@ -160,10 +168,14 @@
 #'  sub-models so that, in these cases, not every row in the tuning parameter
 #'  grid has a separate R object associated with it.
 #'
+#' Finally, it is a good idea to include calls to [require()] for packages that
+#' are used in the function. This helps prevent failures when using parallel
+#' processing.
+#'
 #' @template case-weights
 #' @template censored-regression
 #'
-#' @examplesIf tune:::should_run_examples(suggests = "kernlab") & rlang::is_installed("splines2")
+#' @examplesIf tune:::should_run_examples(suggests = "kernlab") & rlang::is_installed("splines2") && !tune:::is_cran_check()
 #' library(recipes)
 #' library(rsample)
 #' library(parsnip)

R/utils.R

@@ -86,8 +86,8 @@ new_bare_tibble <- function(x, ..., class = character()) {
 # `.iter = 0`.
 .config_to_.iter <- function(.config) {
   .iter <- .config
-  nonzero <- grepl("Iter", .iter)
-  .iter <- ifelse(nonzero, gsub("Iter", "", .iter), "0")
+  nonzero <- grepl("^[iI]ter", .iter)
+  .iter <- ifelse(nonzero, gsub("^[iI]ter", "", .iter), "0")
   .iter <- as.numeric(.iter)
   .iter
 }