Optimal approach for detecting leaf like shapes in opencv

Question

I started to play around with OpenCV for a project I am working on.
For this project I am growing plants in pots. Static pictures of different developmental stages are taken from above (see figure below).

The goal is to detect and ultimately measure (i.e. size) the leaves of the growing plants during different developmental stages.
I tried a naive approach of using Canny edge detecting, however it doesn't work that well (see figure below) because it also detects small edges in the soil (no matter what kind of threshold I used).

I think the better approach is to first segment the image by color and then use some edge detection algorithm to detect the leaves.
Is there a better way to do it?

The leaf structure is known beforehand. Could I use machine learning/classification algorithm to obtain even better results?

I also haven't thought about how to measure the size of the leaves? Are there any common patterns for measuring size and other descriptors? (maybe have a reference object with known size in the picture?).

Finally I also have to deal with occlusion to some extend. This is not visible in the pictures but in later development stages I might have to deal with overlapping leaves. Are there any approaches to deal with that?

I can't bias the picture in my favor (i.e. blacking out the soil, etc) as there might be thousands of plants which have to be processed.

To summarize my questions:

• What's the best approach to detect leafs in a soil (see figures) ?
• Can Machine learning algorithm improve detection?
• How can I measure the size of the leaves?
• How to deal with occlusion/overlapping leaves?

I would be really thankful for some pointers or ideas.

Update (based on Jeff7 comments):

I first ran the mean shift color segmentation together with a floodfill algorithm and ended up with this picture:

When I now run the canny edge detection + findcontours on the that picture the results are much better:

Answer 1

This is an active area of research. I recommend the following papers:

• Scharr et al. 2016: Leaf segmentation in plant phenotyping: a collation study (pdf)

• Bell and Dee 2016: Watching plants grow–a position paper on computer vision and Arabidopsis thaliana. http://doi.org/10.1049/iet-cvi.2016.0127

PlantCV has some facilities for segmenting leaves with distance transforms and watersheds, and I would like to add more. See our preprint, a revised version of which will be published in PeerJ soon.

Answer 2

PlantCV points out that the more you control your setup, the less work you have to do in the software.
If you control your seeding position (rather than just scattering them over the tray), you can mask out around the seedlings and get rid of background clutter. This is also a significant help for working with individual plants, and finding the same plant in different images. Overlapping plants will make for an impossible situation, either when you scatter seed and the sprout next to each other, or when they exceed the size of your spacing. You have to decide if you are planting for the best observations, or optimal growth. The two are not mutually compatible. In my experience, you are correct to use color (cv2.inRange)as the main step, then move on to Canny edge detection. From there, you can get contours. I am guessing you are using Arabadopsis, so you might look for a circular pattern to count leaves (doesn't work with lettuce). Once you have separated your plants and have (outer) contours, you can then use cv2.contourArea and cv2.minAreaRectangle to get some basic metrics. This is the route I am working.

Answer 3

Take a look at mean-shift color segmentation (there's an example included with OpenCV in the samples directory). You can use this to separate your image into 2 classes (plant and soil) and use that to further process your data.

As for measurement, you may want to ignore occlusion effects and camera calibration initially and just look at portion of area in image that is the plant class.

If you want to get down to measuring individual leafs, you may take a "tracking" approach where you use the temporal information as well as the spatial information in the image. The temporal information may be the location and size of the leaf in the previous image. There's probably lots of techniques you could apply, but I'd start simple if I were you and see how far it gets you.

Answer 4

Since you have a great deal of control over the conditions in which you take images, bias matters in your favour. Make a mask out of black card to place around the root of the plant when you want to capture an image. Your problem reduces to one of spotting green pixels against a black background. Since you also control the position of the camera wrt the position of the plant, you should be able to arrange matters such that the pixel/mm ratio is constant across your series of images. Leaf area is then a simple matter of counting pixels.

This doesn't, yet, deal with the issue of occlusion. You could probably figure something out by taking 2 further images, elevations of the plant in 2 orthogonal planes (again using the black background) to get some idea of the shape of the plant.

EDIT after comment ...

Well, your question included the statement 'For this project I am growing a specific plant in a pot' and now you want to process thousands of the damn things. I'd still pursue the possibilities of biasing the image in your favour. For instance, if the image you posted didn't have green plastic in the background then you'd probably have a picture which could be separated, with a good degree of accuracy, by simple thresholding on the green channel of your image(s). So get rid of the green background, and water the soil before imaging to increase it's blackness.

As for the problem of occlusion, you're going to need something smarter than my original suggestion to deal with thousands of plants. Perhaps you could sacrifice a few plants at each stage, measure the 'leaf area observed from directly overhead' then rip off the leaves, arrange them separately on a piece of black card and derive an empirical relationship between total area and observable area.

FURTHER EDITING

OK, so you can't bias the scene in your favour. Have you thought of using a filter on the camera to admit only green light ? Or illumination which makes green objects brighter than not-green objects ? I'm out of ideas on this one ...

FINAL EDIT

I've run out of ideas. I think that your original approach, using colour to discriminate between leaves and background, is good. Since you know the structure of the leaves you might try template matching, but you can estimate both areas and lengths (or differences in area and length) just by counting pixels. You might want to investigate morphological operations (eg skeletonisation) for deriving shape measures from the images. You may find material in the literature on remote sensing of foliage (etc) that helps.

I get the impression that you have mentally committed to an objective of implementing a computer vision system, where your objective is really to monitor the development of the plants and that some of your ideas (eg edge-detection, machine learning) do not contribute to reaching your proper objective.