Deep Learning on the Rocky Shore

The rocky intertidal zone is an area of high biodiversity, and mussel beds are a critical part of this ecosystem—creating habitat and providing an important food source for a variety of animals, from humans to birds to sea stars. 

Climate change and sea level rise are subjecting these habitats to increasing stress, however, and there is a growing need to assess the health and resilience of the blue mussels (Mytilus edulis) and California mussels (Mytilus californianus) that make them up.

Hakai has been monitoring mussel beds on the Central Coast for nearly a decade, via nearshore field studies performed in situ by researchers, and through annual drone surveys. The Institute has been a leader in deploying drones to extend observational reach, allowing researchers to assess sites that are too dangerous or difficult to access. 

Traditional methods for processing drone imagery data, however, are time-consuming and expensive. The use of deep neural networks—designed for processing imagery—has been a major leap forward in terms of enabling Hakai researchers to quickly process and analyze vast amounts of imagery. 

But how does it all work? And what is a neural network, anyway? Tula Quarterly sat down with Taylor Denouden and Will McInnes, both members of the Hakai Institute geospatial team, to learn how the job gets done.

What are your respective backgrounds in working with these kinds of AI processes? 

Taylor: I did my master’s degree at Ontario’s University of Waterloo on computer vision systems for autonomous vehicles, which meant a lot of machine learning training as well as research in that area. After grad school, I came to Hakai and took on a project to detect kelp beds in drone imagery, and then later on, using aerial imagery. And so this mussel-mapping work is an extension of that, where we’re looking at mussel beds on rocks rather than kelp in water.

Will: I've been working in the remote sensing and GIS fields for over a decade. I did my training at the University of Calgary with satellite remote sensing and airborne systems, like aircraft, with traditional remote sensing techniques, and doing classification and time series analysis. The last few years we've been using drones at Hakai, and the AI thing is new to me, so it’s been quite interesting to learn that process. So I know a lot about the other techniques, mainly the drone side of things, and I’ve been learning a lot about how to implement the AI part of it. 

From a standpoint of efficiency, it seems like artificial intelligence is offering us a quantum leap in terms of processing these visuals. 

Will: Yeah, absolutely, We're now talking about being able to do quantification of species over large areas in a matter of minutes, rather than the hours or days it would take to process individual scenes by hand. 

How does the drone process work, that initial gathering of imagery? 

Will: It’s always done by an operator or team who is keeping the drone in their line of sight. For some of the more remote locations, we're on a boat, and we launch the drone from out on deck. On a good weather day, the drone can be up to a kilometer away. It’s been set on autopilot, and it's following an automated pattern, taking pictures that get overlapped and put together in software back at home—creating stitched-together images that can be fed into the machine learning model. We’ll get between half a square kilometer to one square kilometer of terrain covered in a day.

Tell us a bit about how these AI processes work once you have the visual data.  

Taylor: The images we have from a survey have red, green, and blue color channels. The AI model “learns” what transformations and combinations of those red, green, and blue pixels add up to being mussels. There are multiple, complicated steps to go from pixel color to something that says “This is a mussel’ or “This is not a mussel.”  

From the start, this technology is based on simple-input information. It learns how to adjust and combine information so that it can discern between mussels and non-mussels, whether that’s a rock, the water, the sky, or whatever. 

Can you explain what is meant by deep learning and neural networks?

Taylor: The terms neural networks and deep learning are pretty well synonymous. A neural network describes the model itself, something that would be similar to linear regression, which is an algorithm that provides a linear relationship between variables. In neural networks, we use non-linear relationships instead of linear ones and stack a bunch of them together to find much more complex relationships. 

Deep learning is the process for finding the best parameters for our model so it can figure out how to identify things. We do this by showing examples of what we want. So we’ll have a bunch of example images showing mussels, and then have images that have been labeled by a human that show the locations of the mussel beds in that image. 

Those outputs, those definitions, are hand-labeled in the old, slow way, but when we have a whole bunch of them, we can show them to our model over and over again and adjust the parameters—so that the model learns to repeat that process without necessarily having to explicitly define how to do that transformation. That’s the “deep learning” part—how it learns to reproduce some process that is hard to describe in words, but that a human can do automatically.

So deep learning is that kind of iterative learning where, by being shown something over and over, it starts to “understand” and be able to identify something?

Taylor: Right. It comes up with some internal representation, or a set of transformations to go from that input to your output. The “deep” part refers to the neural network itself. Neural networks have a number of layers of transformations, and the deep part just means there’s many of them. And the “learning” part refers to that iterative process.

If one of us is looking at an image, we might think that anything that’s sort of dark and shiny and round is probably a mussel, but the model figures out how to come up with those descriptors itself, without you having to explicitly state them, because there are other things that are black and shiny and round as well that aren’t mussels. The model tries to find a more precise set of descriptors to figure out how to tell those things apart.

And you don’t have to tell it that, right? In other words, that is where the artificial intelligence aspect of this comes in? 

Taylor: That’s the really powerful part of all of this, that it finds the most useful features itself—it finds millions and millions of them and different combinations of them, something that would be pretty impractical to do manually.

To humans on the outside of these operations, is it sort of a black box how it all works? It seems that after you set these AI processes in motion, you are getting all these results and looking at the output and basically saying, “This is great. We don’t know how it’s doing it, but it’s doing a hell of a job.”

Taylor: It’s largely a black box of mathematical operations. We can’t do much more than have the model tell us what part of the image it was looking at when it came up with decisions. There’s a lot of research looking into that, because we would love to know how it’s making decisions so that we can validate them, but that kind of machine learning research is still in its adolescence.

What is most exciting to you about this project, with this application of AI into mussel mapping? 

Will: It’s exciting because it expands our possibilities for collecting scientific data quickly. It’s not the same type of data that we get when we send our crews out into the field and they’re on their knees looking at individual plants and animals and measuring and weighing them, but it gives us an amazing way to survey quickly. 

Right now, we’re working on gooseneck barnacles to add that in as part of the modeling, and that’s an interesting one, because those have been too difficult or dangerous for our teams to study well, since they’re located on exposed outer rocks. But we can get imagery with the drones, and look at dynamics of how they’re interacting with mussels and sea stars and all sorts of stuff. Seagrass will be another one.

Taylor: I don’t know how much I would add to that. I’d only agree that it’s impressive just how much faster these processes are getting, and that it’s saving people a lot of time that would have been spent on boring manual tracing work. This allows them to focus their efforts on more interesting research questions.

Regarding seagrass, that is something that we’ve looked at, but there’s a big challenge in terms of seagrass being covered in water a lot of the time. Our images can’t differentiate very well between submerged seagrass or dark-colored mud. And so it might be interesting to start exploring other kinds of sensors, things like four-band multispectral imagery, which includes near-infrared, or some other specialized camera that could help figure out what things are underwater—which right now we can’t do with a regular image.