In this paper, a simple user-aided, robust, and high-quality fast shadow removal method is proposed. Firstly, an instant learning method is used to remove the shadow of an image, guided by input from the user of two rough areas of shadow and lighting pixels from which the initial shadow mask for shadow removal is derived. Secondly, the detected shadow image is offset and corrected to eliminate the influence of ambient light by a simple input of a color sampling line indicating the rate of change of illumination of the shadow region, penumbra region, and luminance region pixels. Clustering and illumination estimation are then performed for the corrected images. Meanwhile, the shadow is removed by calculating the statistical distribution of the shadow-free regions, and the lighting is restored by estimating the illumination scaling factor. Finally, the energy function used for illumination optimization can perform smooth estimation and image enhancement of the area after shadow removal. Experiments show that our proposed algorithm can generate high-quality shadow-free images and deal with difficult scenes with various texture types and uneven shadows.