Supervised and Semi-Supervised Learning in Vision using Deep Neural Networks

Deep learning has been a huge success in different visiontasks like classification, object detection, segmentation etc.,allowing to start from raw pixels to integrated deep neural models. This thesis aims to solve some of these visionproblems using several deep neural network architecturesin different ways. The first and the core part of this thesisfocuses on a learning framework that extends the previouswork on Semantic Based Regularization (SBR) to integrateprior knowledge into deep learners. Deep neural networksare empirical learners and therefore heavily depend on labeled examples, whereas knowledge based learners on theother hand are not very efficient in solving complex visionproblems. Therefore, SBR is designed as a semi-supervisedframework that can tightly integrate empirical learners withany available background knowledge to get the advantagesof learning from both perception and reasoning/knowledge.The framework is learner agnostic and any learning machinery can be used. In the earlier works of SBR, kernel machinesor shallow networks were used as learners. The approachof the problem, concept of using multi-task logic functionsare borrowed form the previous works of SBR. But for thefirst time, in this research work, the integration of logic constraints is done with deep neural networks. The thesis defines a novel back propagation schema for optimization ofdeep neural networks in SBR and also uses several heuristics to integrate convex and concave logic constraints intothe deep learners. It also focuses on extensive experimental evaluations performed on multiple image classificationdatasets to show how the integration of the prior knowledgein deep learners can be used to boost the accuracy of several neural architectures over their individual counterparts.SBR is also used in a video classification problem to automatically annotate surgical and non-surgical tools from videos ofcataracts surgery. This framework achieves a high accuracy compared to the human annotators and the state-of-the-artDResSys by enforcing temporal consistency among the consecutive video frames using prior knowledge in deep neural networks through collective classification during the inference time. DResSys, an ensemble of deep convolutionalneural networks and a Markov Random Field based framework (CNN-MRF) is used, whereas SBR replaces the MRFgraph with logical constraints for enforcing a regularizationin the temporal domain. Therefore, SBR and DResSys, twodeep learning based frameworks discussed in this thesis, areable to distill prior knowledge into deep neural networks andhence become useful tools for decision support during interoperative cataract surgeries, in report generation, in surgical training etc. Therefore, the first part of the thesis designs scientific frameworks that enable exploiting the wealthof domain knowledge and integrate it with deep convolutional neural networks for solving many real world visionproblems and can be used in several industrial applications.In the present world, a range of different businesses possess huge databases with visuals which are difficult to manage and make use of. Since they may not have an effectivemethod to make sense of all the visual data, it might end upuncategorized and useless. If a visual database does not contain meta data about the images or videos, categorizing it, isa huge hassle. Classification of images and videos throughuseful domain information using these unified frameworkslike SBR is a key solution. The second part of the thesis focuses on another vision problem of image segmentation andthis part of the thesis is more application-specific. However,it can still be viewed as utilizing some universal and basicdomain knowledge techniques with deep learning models.It designs two deep learning based frameworks and makesa head to head comparison of the two approaches in termsof speed, efficiency and cost. The frameworks are built forautomatic segmentation and classification of contaminantsfor cleanliness analysis in automobile, aerospace or manufacturing industries. The frameworks are designed to meet theforemost industry requirement of having an end-to-end solution that is cheap, reliable, fast and accurate in comparisonto the traditional techniques presently used in the contaminant analysis and quality control process. These end-to-endsolutions when integrated with the simple optical acquisitionsystems, will help in replacing the expensive slow systemspresently existing in the market.