Introduction
The global fishing industry has experienced decreased production mainly due to depleted fishery resources and the strengthening of international fishing standards. In response, fish production by the aquaculture industry has steadily increased to provide alternative fish resources. Despite these increases, the Korean aquaculture industry has shown stagnant growth due to its manual labour-intensive farming techniques. To overcome this problem and create a more systematic aquaculture management system that is compatible with the oncoming fourth industrial evolution, it is important to automate some aspects of the aquaculture industry through innovations such as smart fisheries and automatic sensors for water temperature and dissolved chemical detection, as well as automatic feeding systems and real-time underwater monitoring. Although many technologies have already been adapted to monitor physical tank and cage conditions, those for automatic monitoring of fish behaviour are still needed for real-time fish condition assessment.
Fish have been used as model animals in behaviour monitoring studies, particularly those exploring image analysis of behaviour data collected by underwater acoustic cameras, with and without external stimulation. Underwater acoustic cameras have been applied to collect fish echo data to analyse the distribution of fishery resources and its effects on fish movement (Jang et al., 2016). Video tracking has also been used to trace moving organisms, on land and in air and water (Maggio & Cavallaro, 2011; Noldus et al., 2001). However, to date, no studies have explored the adaptation of automatic real-time monitoring systems that use artificial intelligence to analyse movement patterns that may indicate the health status or abnormal conditions of fish in aquaculture systems.
Deep learning is part of a broader family of machine learning methods based on artificial neural networks (Bengio et al., 2013; LeCun et al., 2015; Schmidhuber, 2015). With recent improvements in computational speed, various algorithms have been exploited to overcome the limitations of deep learning for real-time applications of video or image data, which are otherwise complex and time-consuming. Convolutional neural network (CNN) is a deep-learning technique that provides rapid analysis of calculations performed using a graphics processing unit (GPU); specialised CNNs have been developed for region extraction (e.g., R-CNN; Girshick et al., 2015) and object detection (e.g., you only look once [YOLO]; Redmon & Farhadi, 2018). The YOLO algorithm simultaneously integrates and processes object detection and recognition patterns, and therefore is suitable for rapid, high-accuracy real-time applications.
Rock bream (Oplegnathus fasciatus) inhabits rocky marine areas in shallow coastal regions of Korea, Japan, Taiwan, and Hawaii (Choi et al., 2002); its economic value is increasing in the Korean aquaculture industry. Despite technological advancements in artificial rock bream seeding (Fukusho, 1979; Kumai, 1984), its aquaculture productivity requires further improvement. Studies have examined the photo-response of rock bream (Jang et al., 2019). Artificial intelligence algorithms, including a deep-learning neural network, have been developed for identifying fish species (Allken et al., 2019). However, to solve problems associated with automated fish health monitoring systems in aquaculture, it is still necessary to develop an algorithm that can detect abnormal fish behaviours, which may require training and validation datasets. This study recorded the swimming patterns of healthy rock bream and those exposed to unhealthy conditions. A deep learning-based algorithm was applied to evaluate rock bream behaviour based on recorded fish movement data.
Materials and Methods
Rock bream O. fasciatus (body weight, ~8.8 ± 1.8 g; body length, 74.0 ± 6.0 mm) were obtained from Ji-Yeon Fisheries (Geoje, Korea). The animal tracking system EthoVision XT10 was from Noldus (Wageningen, the Netherlands). We purchased the anaesthetic 2-phenoxyethanol from Sigma-Aldrich (St. Louis, MO, USA) and obtained a charge-coupled device (CCD) camera from Samsung Electronics (Suwon, Korea). The YOLO v3 algorithm, which is based on a Darknet-53 CNN backbone, was used to analyse fish behaviour.
Rock bream were maintained in a 2-ton circulation tank prior to behaviour monitoring. The breeding water was partially replaced with sand-filtered, aerated seawater (salinity, 33 ± 0.5 psu; pH 7.4 ± 0.7). Rock bream were acclimated for 1 week in a rectangular water tank (1 m × 0.45 m, natural seawater, 18°C–20°C). The water salinity experiment was conducted in a 1 m × 0.6 m tank with an inlet and outlet for exchanging water. After acclimatisation, fish movement was recorded by a CCD camera positioned at the top of the tank (1 m above the water surface), first for a single fish and then for a group of five fish. We induced abnormal rock bream behaviour using two methods: adding an anaesthetic (2-phenoxyethanol; 0.05% v/v) to the water tank and changing the tank water from seawater to fresh water. A complete change from seawater to fresh water was achieved in 20 min using the tank inlet and outlet. In each experiment, we recorded fish behaviour for 15–30 min after inducing the abnormal behaviour. The housing and maintenance of the fish conformed to the regulations of The Institutional Animal Care and Use Committee of Pukyong National University (Busan, Korea).
The collected data included 10,110 rock bream images obtained from the video recording. Among these, we randomly selected 210 images, and divided these into a training dataset containing 168 images and a test dataset containing 42 images. No images were included in both datasets.
In pre-processing, the rock bream images were adjusted to a standard size of 416 × 416 × 3 for analysis by the YOLO algorithm. Images of rock bream swimming upright and lying on their side were considered to exhibit normal and abnormal swimming, respectively. The input images were entered into the pretrained CNN, which predicted bounding box information including the location and size of the detected object in the input image for target detection. Within a total learning time of 2 h, class-specific bounding boxes were produced for all input images (Fig. 1).
The ability of the algorithm to detect rock bream displaying abnormal behaviour was evaluated based on the training data, based on the rates of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in a confusion matrix. The pixel accuracy, precision, and recall were calculated as follows:
Results and Discussion
Behaviour analysis is important for confirming the health of target fish in aquaculture. In this study, images of swimming rock bream, one of the main aquaculture species in Korea, were used to explore the application of an artificial intelligence algorithm to analysing images for species identification and detecting abnormal swimming behaviour. This used a camera located in a fixed position on top of a tank containing one rock bream at a time. Most of the fish swam upright (Fig. 2). In addition to image analysis for species identification using artificial intelligence (see below), the speed of the rock bream was calculated by tracing their movements before and after adding an anaesthetic or abruptly changing the water salinity. Healthy rock bream moved at an average speed of 3.8 ± 0.6 cm/s, but after 2-phenoxyethanol (2-PE) treatment their speed initially increased to 16.5 ± 3.0 cm/s, and then they gradually slowed and finally stopped and lay on their sides (Fig. 3). The slightly faster speed (Fig. 2b) observed in the water salinity experiment might result from the larger tank used for this experiment. The condition-dependent change was also observed in images of rock bream taken from a fixed position, which involved a transition from a narrower top view of the fish under normal conditions, to a wider lateral view of the rock bream following the 2-PE or low salinity treatments (Fig. 3 insets). Since these behaviours mimic those of rock bream in poor condition in aquafarms, these images can be used as training data for evaluating the suitability of the deep-learning algorithm for detecting unhealthy aquafarm conditions.
Artificial intelligence has been applied to many industries, including aquaculture. In this study, images of a swimming rock bream were entered into a pretrained CNN for target detection. Fig. 2 shows that rock bream can be clearly identified from images of one or many individuals in a tank using the CNN-based YOLO algorithm. The application of artificial intelligence was extended to detect a disastrous situation in an aquaculture facility by analysing the swimming patterns of rock bream. This used images of rock bream swimming upright or lying on their sides as training data for normal and abnormal situations, respectively. In each iteration of the training process, class-specific bounding boxes were produced for each image, and the count information was updated. Within a total learning time of 2 h, the algorithm successfully learned to discriminate normal and abnormal swimming behaviour (Fig. 4). We also evaluated the training process of the YOLO algorithm through loss-of-function analysis. Loss function is the current learning indicator of deep-learning models and its value should be minimised as the number of lessons increases. Loss was minimised within 40 learning iterations (Fig. 5) with its value converging on zero. The result confirmed the suitability of the algorithm for detecting unhealthy aquafarm conditions. We further evaluated the performance of the YOLO algorithm for detecting abnormal rock bream behaviour using a confusion matrix based on the relationship between actual measurements and predicted results with data described as TP, TN, FP, and FN. Based on the learned data, the confusion matrix of abnormal rock bream detected showed TP = 78, FP = 0, FN = 2, and TN = 25 for the normal conditions and TP = 25, FP = 2, FN = 0, and TN = 78 for the abnormal conditions (Table 1). From these data, we calculated the accuracy (0.981), precision (0.963), recall (0.988), and F1 score (0.974) of the algorithm. Together, these results indicate excellent algorithm performance.
| Abnormal class | Actual behaviour | ||
|---|---|---|---|
| Abnormal | Normal | ||
| Predicted behaviour | Abnormal | 25 (TP) | 2 (FP) |
| Normal | 0 (FN) | 78 (TN) | |
The development of artificial intelligence-based technologies suitable for application in aquaculture has the potential to transform this manual labour-intensive industry to rely on more efficient automated systems (Yang et al., 2020). Various deep-learning algorithms have been applied in marine biology for the development of smart fisheries, including automated fish detection based on cascade classification of Haar-like features in underwater images captured by a remotely operated vehicle in an ocean survey (Cutter et al., 2015) and deep learning-based fish detection (Levy et al., 2018). Recent deep learning-based studies have focused on detecting fish species using CNNs (Han et al., 2020), classifying goldfish species using pruned neural networks (Ayob et al., 2021), and detecting sea cucumber using a single-shot multibox detector (Ma et al., 2019). This study also proved that rock bream can be identified using an artificial intelligence-based algorithm.
In aquaculture farms, real-time fish monitoring would allow machines to automatically detect deteriorating fish condition and trigger an alarm before serious damage could occur. To develop an artificial intelligence-based algorithm for identifying abnormal situations in aquaculture, it is critical to secure learning data corresponding to each situation. This study obtained behaviour datasets of rock bream under abnormal situations using anaesthetic or low-salinity treatments. Abnormal behaviors of fish under harsh conditions include an irregular movement, reduced food intake, increased surfacing behaviors, changes in the frequency of gill opening, sluggish swimming, delayed response, loss of vertical equilibrium, departure from crowds, and scrubbing of the tank floor. Changes in swimming speed is another criterion used to define abnormal behavior in fish. This was shown the phenomenon that, immediately after administration of the anesthetic, the speed of the fish increased rapidly and then decreased and eventually stops. Among these behavioral characteristics, the flipping phenomenon of the fish, one of the simplest indicators, was used to identify the abnormal behavior in this study. The study results indicate that a deep-learning algorithm can be used to detect abnormal swimming under aquaculture conditions after learning representative patterns of normal and abnormal swimming behaviour. The accuracy of the proposed algorithm for detecting abnormal swimming by anaesthetic-treated rock bream was 98.1%. Our findings suggest that deep learning has the potential to improve aquafarm efficiency and prevent economic losses through automated monitoring of fish.
