Reminder of the First Paper on Transfer Learning in Neural Networks, 1976
DOI:
https://doi.org/10.31449/inf.v44i3.2828Abstract
This paper describes a work on transfer learning in neural networks carried out in 1970s and early 1980s, which produced its first publication in 1976. In the contemporary research on transfer learning there is a belief that pioneering work on transfer learning took place in early 1990s, and this paper updates that knowledge, pointing out that the transfer learning research started more than a decade earlier. This paper reviews that 1970s research and addresses important issues relevant for the current transfer learning research. It gives a mathematical model and geometric interpretation of transfer learning, and a measure of transfer learning indicating positive, negative, and no transfer learning. It presents experimental investigation in the mentioned types of transfer learning. And it gives an application of transfer learning in pattern recognition using datasets of images.References
A. Agarwal, R. Mammone, and D. Naik (1992) An on-line training algorithm to overcome catastrophic forgetting. In Intelligence Engineering Systems through Artificial Neural Networks. volume 2, pages 239-244. The American Society of Mechanical Engineers, AS~IE Press.
J. Baxter, R. Caruana, T. Mitchell, L. Pratt, D. Silver, S. Thrun (organizers) Learning to Learn: Knowledge Consolidation and Transfer in Inductive Systems, NIPS*95 Post-conference workshop, Vail, Colorado http://socrates.acadiau.ca/courses/comp/ dsilver/NIPS95ltl.nips95.workshop.pdf
S. Bozinovski (1972) Perceptrons: Training in pattern recognition. (original in Croatian: Perceptroni i obucavanje u prepoznavanju oblika) unpublished student scientific competition paper, University of Zagreb
S. Bozinovski (1974). Perceptrons and possibility of simulation of a teaching process (original in Croatian: Perceptroni i mogucnost simuliranja procesa obucavanja), unpublished M.Sc. thesis, Electrical Engineering Department, University of Zagreb
S. Bozinovski, A. Fulgosi (1976). The influence of pattern similarity and transfer of learning upon training of a base perceptron B2. (original in Croatian: Utjecaj slicnosti likova i transfera ucenja na obucavanje baznog perceptrona B2), Proc. Symp. Informatica 3-121-5, Bled.
S. Bozinovski, A. Santic, A. Fulgosi (1977). Normal teaching strategy in pair-association in the case teacher:human-learner:machine. (original in Croatian: Normalna strategija obicavanja u obucanju asocojacije parova u slucaju ucitelj:covjek-ucenik:masina), Proc. Conf. ETAN, 21:IV-341-346, Banja Luka, [available online].
S. Bozinovski (1978). Experiments with non-biological systems teaching. (original in Macedonian: Eksperimenti na obucuvanje na nebioloski sistemi) Proc. Conf ETAN, 22:IV-371-379, Zadar [available online].
S. Bozinovski (1981). Teaching space: A representation concept for adaptive pattern classification. COINS Technical Report, University of Massachusetts at Amherst, No 81-28 [available online].
S. Bozinovski (1985a). Adaptation and training: A viewpoint. Automatika 26 (3-4) 137-144
S. Bozinovski (1985b). A representation theorem for linear pattern classifier training. IEEE Transactions on Systems, Man, and Cybernetics 15(1): 159-161
S. Bozinovski (1995). Neuro-genetic agents and a structural theory of self-reinforcement learning systems. CMPSCI Technical Report 95-107, University of Massachusetts at Amherst [available online].
K. Fukushima (1975) Cognitron: A self organizing multilayered neural network. Biological Cybernetics 20: 121-136
K. Fukushima (1980) Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics 36: 193-202
V. Glushkov (1967) Introduction to Cybernetics (original in Serbian: Uvod u Kibernetiku, translated from Russian, published by Zavod za izdavanje udzbenika Srbije)
I. Goodfellow, Y. Bengio, A. Courville, (2016) Deep Learning, MIT Press
M. McCloskey, N. Cohen (1989). Catastrophic interference in connectionist networks: the sequential learning problem. The Psychology of Learning and Motivation, 24
M. Minsky, S. Papert (1969) Perceptrons. The MIT Press, 1969
D. Naik, R. Mammone (1993) Learning by learning in neural networks, In R. Mammone (ed.) Artificial Neural Networks for Speech and Vision, Chapman and Hall, London.
S. Pan, Q. Yang (2010). A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10), 1345– 1359
L. Pratt, J. Mostow, C. Kamm (1991). Direct transfer of learned information among neural networks. In Proceedings of the Ninth National Conference on Artificial Intelligence (AAAI-91), p. 584-589, Anaheim, CA.
L. Pratt (1993). Discriminability-based transfer between neural networks. In NIPS Conference: Advances in Neural Information Processing Systems 5 Morgan Kaufmann Publishers. pp. 204-211
L. Pratt, B. Jennings (1996) A Survey of Transfer Between Connectionist Networks, Connection Science 8(2) 163-184.
F. Rosenblatt (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review 65: 386-408
F. Rosenblatt (1962). Principles of Neurodynamics. Spartan Books
D. Rumelhart, J. McClelland, and the PDP Group (1986). Parallel Distributed Processing. MIT Press.
N. Sharkey and A. Sharkey (1992) Adaptive generalisation and the transfer of knowledge, Proceedings of the Second Irish Neural Networks Conference, Belfast.
C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu (2018). A Survey on Deep Transfer Learning, arXiv:1808.01974v1 [cs.LG] 6 Aug 2018.
S. Thrun, T. Mitchell (1993) Lifelong robot learning, Technical Report IAI-TR-93-7, Institute for Informatics III, University of Bonn.
H. Wang, C. Li, X. Zhen, W. Yang, B. Zhang (2019) Gaussian Transfer Convolutional Neural Networks, IEEE Transactions on Emerging Topics in Computational Intelligence 3 (5) 360-368.
K. Weiss, T. Khoshgoftaar, D. Wang (2016) A survey of transfer learning. Journal of Big Data 3:9.
Wikipedia > Transfer Learning (June 2020) https://en.wikipedia.org /wiki/Transfer_learning
Downloads
Published
How to Cite
Issue
Section
License
I assign to Informatica, An International Journal of Computing and Informatics ("Journal") the copyright in the manuscript identified above and any additional material (figures, tables, illustrations, software or other information intended for publication) submitted as part of or as a supplement to the manuscript ("Paper") in all forms and media throughout the world, in all languages, for the full term of copyright, effective when and if the article is accepted for publication. This transfer includes the right to reproduce and/or to distribute the Paper to other journals or digital libraries in electronic and online forms and systems.
I understand that I retain the rights to use the pre-prints, off-prints, accepted manuscript and published journal Paper for personal use, scholarly purposes and internal institutional use.
In certain cases, I can ask for retaining the publishing rights of the Paper. The Journal can permit or deny the request for publishing rights, to which I fully agree.
I declare that the submitted Paper is original, has been written by the stated authors and has not been published elsewhere nor is currently being considered for publication by any other journal and will not be submitted for such review while under review by this Journal. The Paper contains no material that violates proprietary rights of any other person or entity. I have obtained written permission from copyright owners for any excerpts from copyrighted works that are included and have credited the sources in my article. I have informed the co-author(s) of the terms of this publishing agreement.
Copyright © Slovenian Society Informatika
