ImageDataGenerator resizing and transforming images before preprocessing

[ldm314]

I am trying to use the preprocessing function to take a network sized crop out of inconsistently sized input images instead of resizing to the network size. I have tried to do this using the preprocessing function but found that it is not easily possible. Using Keras 2.2.2

• ImageDataGenerator does not accept None as a type for target_size which should cause load_img to not resize things.

C:\ProgramData\Anaconda3\envs\tensorflow\lib\site-packages\keras_preprocessing\image.py in __init__(self, directory, image_data_generator, target_size, color_mode, classes, class_mode, batch_size, shuffle, seed, data_format, save_to_dir, save_prefix, save_format, follow_links, subset, interpolation)
   1665         self.directory = directory
   1666         self.image_data_generator = image_data_generator
-> 1667         self.target_size = tuple(target_size)
   1668         if color_mode not in {'rgb', 'rgba', 'grayscale'}:
   1669             raise ValueError('Invalid color mode:', color_mode,

TypeError: 'NoneType' object is not iterable

• To accomplish my goal, I modified the image data generator to pass none to load_img and then to resize afterwards if the size doesn't match the target. This hack works for my scenario:

    def _get_batches_of_transformed_samples(self, index_array):
        batch_x = np.zeros(
            (len(index_array),) + self.image_shape,
            dtype=backend.floatx())
        # build batch of image data
        for i, j in enumerate(index_array):
            fname = self.filenames[j]
            img = load_img(os.path.join(self.directory, fname),
                           color_mode=self.color_mode,
                           target_size=None)
            x = img_to_array(img, data_format=self.data_format)
            # Pillow images should be closed after `load_img`,
            # but not PIL images.
            if hasattr(img, 'close'):
                img.close()
            params = self.image_data_generator.get_random_transform(x.shape)
            x = self.image_data_generator.apply_transform(x, params)
            x = self.image_data_generator.standardize(x)
            width_height_tuple = (self.target_size[1], self.target_size[0])
            if (x.shape[1],x.shape[0]) != width_height_tuple:
              x=cv2.resize(x,width_height_tuple, interpolation=cv2.INTER_AREA)
            batch_x[i] = x

While looking into this I saw that the preprocessing function runs at the start of standardize, which is after the random transforms are applied. To me this sounds like preprocssing is a bad name since it isn't actually happening first.

[TurnipEntropy]

Agreed; Having the same problem. I'm resizing the images in the preprocessing function, but then batch_x[i] complains about the image being the wrong size. No, it's just using self.x.shape (older version, admittedly) or self.image_shape instead of determining the image shape from the images flowing into it. I do understand the dilemma here though. You don't want to run the batch process before setting self.image_shape, and you don't want to allow different batches to have different image sizes. It's tricky. Maybe if target_size were in flow, this problem could be avoided though.

[rragundez]

I would like to help you guys out. If I understand correctly you need to be able to crop images instead of resizing/stretching/etc? I just want to get the point clear.

[ldm314]

@rragundez, The specific need is the ability to resize the images in the preprocessing function. In the end, I wrote a generator specific to the application.

[pratiklodha95]

@rragundez I jumped into the issue which you described where I want to crop the image and before other transformation in the images are done

[Dref360]

I think this will be solved in the new design?
See https://github.com/keras-team/governance/pull/6/files

I know that this proposition is not yet accepted, but once it is, it should be pretty fast.

[zaccharieramzi]

I also find myself in the position where I would like to flow directly from a directory where images have different sizes because I can't hold all the images in memory and also would like to crop/rotate/pad on-the-fly.

Currently I would need to implement my own Sequence object, but if there was a resizing function that could be called before the pre-processing function (that is supposed to return images of the same size as the input) and before the augmentation, I could just use that.

The new design @Dref360 pointed out was closed I think sine die. I also think in this new design the pre-processing function (even if called before the resizing and augmentation) was supposed to keep the same size for input and output.

Wouldn't it be possible to just add the possibility to "override" (i.e. provide our own) the resizing function when flowing from a directory? This wouldn't change the preprocessing function's behaviour and I guess (didn't look at the code for this) would mean just replacing the current resizing function if overridden. I can try and do a PR for this.

[zaccharieramzi]

For those interested by this problem, I have a fork where I implemented this idea, I have been using it and it's working great so far.
I opened a PR, but so far no one has taken the time to review it: #248.