Gulp outputs that series has finished, but processes still running

[paul-uz]

I am running a few tasks via series, with one task scanning some directories, and another zipping the contents of each directory.

Despite the console displaying the message Finished 'cleanAndBuild' after 28 s this isn't actually the case, with the console not exiting until much later, at about 120s.

gulpfile.js

const { exec } = require('child_process');
const { series, dest, src } = require('gulp');
const del = require('del');
const fs = require('fs');
const zip = require('gulp-zip');

function clean() {
  return del('.dist/');
}

function installPackages() {
  const directories = fs.readdirSync('lambda/');
  directories.forEach((dir) => {
    if (fs.existsSync('lambda/' + dir + '/package.json')) {
      console.log('package.json exists in ' + dir);
      const path = `lambda/${dir}/`
      exec(`npm --prefix ${path} ci ${path} --production`)
    }
  });
  return Promise.resolve();
}

function buildZip() {
  console.log('zipping...');
  const directories = fs.readdirSync('lambda/');
  directories.forEach((dir) => {
    return src('lambda/'+dir+'/**').pipe(zip(`${dir}.zip`)).pipe(dest('.dist/lambda/'+dir));
  });

  return Promise.resolve();
}

exports.cleanAndBuild = series(clean, installPackages, buildZip);

Using the CLI tool gnomon I was able to get the true timing of running my task.

Why is gulp saying cleanAndBuild has finished when it hasn't? And is there anything I can do to improve this script to make it faster/more efficient?

Thanks

[phated]

You aren't writing your gulp file correctly. This isn't a support forum, so I'll transfer this to GitHub discussions. You probably want to review our getting started guide to understand asynchronousity

[paul-uz]

@phated could you point out what I'm doing wrong?

I also tried pushing the exec() and src() calls into an array and using Promise.all() but it seemed that the exec command at least was running regardless of being called with Promise.all() and I couldn't understand why.

The gulp documentation isn't very helpful if I'm honest

[begin-again]

you want to create pipelines using the gulp methods. So instead of fs.readdirSync use gulp.src with a glob pattern that fits your use case.

[paul-uz]

@begin-again the problem is, src and dest are too "simple"

There is semeingly no way to pass variables into dest to dynamically change the output directory.

Here is my scenario:

/root
    /foo
    /bar
    /baz

I need to find each directory inside root, zip the contents, and drop them into a new directory like below

/dist
    /foo
        zip.zip
    /bar
        zip.zip
    /baz
        zip.zip

But I am unable to find any information on how to create the foo, bar and baz directories in the dist directory, when using dest()

I tried using a glob string in the dest() call, but it literally just makes directories with a name of * or **. I want to be able to modify the output based on the current directory that matches the src() call.

[begin-again]

You can create multiple streams and return them as a single stream.

Some plugins in use here are: gulp-if, merge-stream, gulp-concat

const copyJS = () => {

    var appStream;
    var allStreams = merge();

    flog('copyJS started');

    Config.applications.forEach( application => {
        flog(`copyJS Starting on ${application.name}`);

        appStream = src( `${Config.path.src}/includes/${application.name}/**/*.js` )
            .pipe(concat('app-bundle.js'))
            .pipe(gulpif(argv.min, uglify(Config.options.ugly).on("error", flog)))
            .pipe(dest( path.join( Config.path.build, 'includes', application.name, path.sep ) ) )
            .on('error', (err) => {
                flog.error(`copyJS on ${application.name} ended`, err.toString());
                process.exit(1);
            })
            .on('end', () => {
                flog(`copyJS on ${application.name} finished`);
            });

        allStreams.add(appStream);
    });

    return allStreams;
};

[paul-uz]

@begin-again hmm, that example is quite different to what I'm trying to achieve, but I'll try to decipher it.

Where is path coming from in the line .pipe(dest( path.join( Config.path.build, 'includes', application.name, path.sep ) ) )?

What is Config.applications exactly?

[begin-again]

@paul-uz I was just trying to show something comparable which I have in use in one of my builders. The creation of a stream of each installPackages looked like a match to me. Also consider promisfying exec with utils.promisify and returning a Promise.all

• applications is an array of strings
• const path = require('path');, yeah that is confusing.
• Config is just an object with some default values that we add to during initialization. The builder is kicked off from a command line script where the user options are also added. I could create a gist of the whole thing but most of the tasks are very much like above.

There are a lot of gulpfiles on github, perhaps you could find a better match.

[paul-uz]

@begin-again I appreciate the help!

So, if I went down a similar route, the first step would be creating a list of the directories I want to zip up? How do you generate your applications array exactly? I need this part to be dynamic, as we add new functions all the time, each in one of those sub-directories.

src() can find the right sub-directories, but can I utilise src() to generate that initial list of sub-directories to then loop through afterwards and create the multi-stream?

In your dest() called, why are you calling path.join()? What value does path output as?