{"id":4719,"date":"2025-02-04T14:36:22","date_gmt":"2025-02-04T19:36:22","guid":{"rendered":"https:\/\/blog.daed.com\/?p=4719"},"modified":"2025-02-26T19:56:41","modified_gmt":"2025-02-27T00:56:41","slug":"the-history-of-neural-networks","status":"publish","type":"post","link":"https:\/\/blog.daed.com\/?p=4719","title":{"rendered":"The History of Neural Networks"},"content":{"rendered":"

<\/h2>\n

Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!<\/span><\/p>\n

<\/p>\n

You may have seen a diagram like the one above when learning about neural networks. But what do those circles and arrows really mean? Let\u2019s take a walk through history to explain.<\/span><\/p>\n

 <\/p>\n

Neural networks, in theory<\/span><\/h2>\n

The theoretical neural network was first proposed by the psychologist-logician duo of Warren McCulloch and Walter Pitts back in 1943 in their landmark article, \u201cA Logical Calculus of the Ideas Immanent to Nervous Activity.\u201d They were interested in modelling the nervous system as a combination of mathematical functions.<\/span><\/p>\n

In the diagram above, the circles represent \u201cneurons,\u201d a single processing unit of a neural network. The neurons store a set of random numbers, called “weights” inside of them. Each neuron receives some input numbers, represented by the pointed ends of each arrow, or \u201csynapse.\u201d The input numbers get multiplied by the weights and added together, and the result of that calculation is the output of the neuron. By hooking many of these neurons together in what we call “layers,” we create a “neural network.”\u00a0<\/span><\/p>\n

So what goes into the network, and what comes out?\u00a0<\/span><\/p>\n

An input is some sort of data, be it an image, a series of numbers, or a string of text. The output will be whatever the ML engineer wants to know about the data, or what the ML engineer wants the neural network to be able to say about the data. For example, an input could be portraits of pets, and the outputs could be \u201ccat\u201d or \u201cdog,\u201d with the neurons representing the <\/span>probability<\/span><\/i> that the photo is of a \u201ccat\u201d or \u201cdog.\u201d<\/span><\/p>\n

 <\/p>\n

Early work on artificial neural networks<\/span><\/h2>\n

This example actually describes one of the first neural networks ever created. A psychologist named Frank Rosenblatt created the \u201cperceptron\u201d in the 1950s, which aimed to classify photos into different categories.<\/span><\/p>\n

\"A
Photo Credit<\/a><\/figcaption><\/figure>\n

How did it work? The neurons in the network became adjustable potentiometers, and the synapses became wires. At the end of the network, the brightest output bulb represented the category that the image had been classified into.\u00a0<\/span><\/p>\n

Rosenblatt \u201ctrained\u201d his perceptron by showing it many inputs and checking the correctness of its output. When the network was wrong, he went back and manually adjusted the weights of the potentiometers along the way to get the network to produce the correct output. After doing this process over and over again, adjusting the weights according to an optimization algorithm called <\/span>stochastic gradient descent<\/span><\/i>, the perceptron was trained to correctly recognize patterns.<\/span><\/p>\n

Other researchers made multilayer perceptrons that contained multiple internal layers of computation trained with SGD. However, research stagnated for decades after this early work on perceptrons.\u00a0<\/span><\/p>\n

In the 1980s, several different researchers, including Paul Werbos and David Rumelhart, incorporated a mathematical technique called <\/span>backpropagation <\/span><\/i>into multilayer perceptron training, which allowed information about whether the network outputted correctly or incorrectly to be incorporated into the interior (hidden) layers during training. The machine could \u201clearn\u201d on its own.<\/span><\/p>\n

 <\/p>\n

Convolution and Deep learning<\/span><\/h2>\n

In the late 1980s and 1990s, a type of neural network designed for image processing called <\/span>convolutional neural networks <\/span><\/i>(CNN) was developed. Yann LeCun trained CNNs using backpropagation to create the series of networks known as LeNet, which excelled at recognizing handwritten numbers. By the end of the decade, CNNs were reading over 10% of all checks in the US.\u00a0<\/span><\/p>\n

However, there were not many other realistic applications of neural networks. The trouble was that constructing a network required careful engineering to take real-life input\u2013photos, financial information, anything\u2013and figure out some way to represent it as numbers to be fed into the neural network.\u00a0<\/span><\/p>\n

Then, advancements in processing for computer vision, such as the spread of Graphical Processing Units (GPUs) from NVIDIA and the availability of large image datasets, led researchers to invest in <\/span>deep learning<\/span><\/i>, meaning neural networks with many hidden layers, which allowed the machine to have specific functions for recognizing small details in data. The machine could incorporate more layers to automatically learn how to represent real-world data as inputs for deeper layers in the network.\u00a0<\/span><\/p>\n

Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton won the ImageNet 2012 competition for their deep convolutional neural network, which kicked off the modern AI boom. Hinton went on to win the 2024 Nobel Prize in Physics along with Yoshua Bengio and Yann LeCun for their work on deep learning.<\/span><\/p>\n

 <\/p>\n

Attention<\/span><\/h2>\n

In 2017, a team of researchers at Google published a paper called \u201cAttention Is All You Need\u201d (Vaswani, et al). They proposed a network called the <\/span>Transformer<\/span><\/i>, which greatly reduced the amount of computation needed to represent input and output for previously difficult-to-manage data, like raw text. Transformers encode texts as small lists of numbers called tokens. On top of this, Transformers factor in the specific placement of tokens within a body of text, which helps the network figure out which tokens are most important for creating a good output.\u00a0<\/span><\/p>\n

The Transformer now underpins most contemporary large language model (LLM) technology, like that developed by OpenAI and Anthropic, as it allows the network to \u201cpay attention\u201d to relevant information that might have been mentioned much earlier. Transformers can now even be applied to non-textual inputs, like images and audio.\u00a0<\/span><\/p>\n

 <\/p>\n

Today<\/span><\/h2>\n

Neural networks have proliferated into many areas of technology and life today, performing simple and complex, broad and specific tasks. In computer vision, models like You Only Look Once (YOLO) allow the quick detection and classification of objects within images. In audio processing, models like Spleeter can separate musical pieces into \u201cstems\u201d, allowing musicians to create instant acapella or instrumentals from existing songs. NASA is using neural networks to identify newly discovered galaxies in data that is decades old.\u00a0<\/span><\/p>\n

From the bulky perceptron, which took up an entire room, we are now able to run neural networks in the cloud or on edge devices like the NVIDIA Jetson. Neural networks have evolved from an academic curiosity into an indispensable tool that continues to reshape how we solve complex problems and interact with technology.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least …<\/p>\n","protected":false},"author":1,"featured_media":4770,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,222],"tags":[],"class_list":["post-4719","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-design-thinking","category-software-engineering"],"yoast_head":"\nThe History of Neural Networks - daed.com<\/title>\n<meta name=\"description\" content=\"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.daed.com\/?p=4719\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The History of Neural Networks - daed.com\" \/>\n<meta property=\"og:description\" content=\"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.daed.com\/?p=4719\" \/>\n<meta property=\"og:site_name\" content=\"daed.com\" \/>\n<meta property=\"article:published_time\" content=\"2025-02-04T19:36:22+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-02-27T00:56:41+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/blog.daed.com\/wp-content\/uploads\/2025\/02\/6.png\" \/>\n\t<meta property=\"og:image:width\" content=\"2000\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Daedalus\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Daedalus\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.daed.com\/?p=4719#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.daed.com\/?p=4719\"},\"author\":{\"name\":\"Daedalus\",\"@id\":\"https:\/\/blog.daed.com\/#\/schema\/person\/ffe3d55f759956aa85792c64b0d0f984\"},\"headline\":\"The History of Neural Networks\",\"datePublished\":\"2025-02-04T19:36:22+00:00\",\"dateModified\":\"2025-02-27T00:56:41+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.daed.com\/?p=4719\"},\"wordCount\":1038,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/blog.daed.com\/#organization\"},\"articleSection\":[\"design thinking\",\"Software Engineering\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.daed.com\/?p=4719#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.daed.com\/?p=4719\",\"url\":\"https:\/\/blog.daed.com\/?p=4719\",\"name\":\"The History of Neural Networks - daed.com\",\"isPartOf\":{\"@id\":\"https:\/\/blog.daed.com\/#website\"},\"datePublished\":\"2025-02-04T19:36:22+00:00\",\"dateModified\":\"2025-02-27T00:56:41+00:00\",\"description\":\"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!\",\"breadcrumb\":{\"@id\":\"https:\/\/blog.daed.com\/?p=4719#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.daed.com\/?p=4719\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.daed.com\/?p=4719#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.daed.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The History of Neural Networks\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.daed.com\/#website\",\"url\":\"https:\/\/blog.daed.com\/\",\"name\":\"daed.com\",\"description\":\"research, design, and engineering thinking\",\"publisher\":{\"@id\":\"https:\/\/blog.daed.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.daed.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/blog.daed.com\/#organization\",\"name\":\"daed.com\",\"url\":\"https:\/\/blog.daed.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.daed.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/blog.daed.com\/wp-content\/uploads\/2019\/10\/White_Daedalus.png\",\"contentUrl\":\"https:\/\/blog.daed.com\/wp-content\/uploads\/2019\/10\/White_Daedalus.png\",\"width\":5249,\"height\":745,\"caption\":\"daed.com\"},\"image\":{\"@id\":\"https:\/\/blog.daed.com\/#\/schema\/logo\/image\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.daed.com\/#\/schema\/person\/ffe3d55f759956aa85792c64b0d0f984\",\"name\":\"Daedalus\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.daed.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8bf29d93ebf6e34ef10f067eac16bcb56fdf8a13ae323fdda0b07b25d15d6c61?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8bf29d93ebf6e34ef10f067eac16bcb56fdf8a13ae323fdda0b07b25d15d6c61?s=96&d=mm&r=g\",\"caption\":\"Daedalus\"},\"url\":\"https:\/\/blog.daed.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The History of Neural Networks - daed.com","description":"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.daed.com\/?p=4719","og_locale":"en_US","og_type":"article","og_title":"The History of Neural Networks - daed.com","og_description":"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!","og_url":"https:\/\/blog.daed.com\/?p=4719","og_site_name":"daed.com","article_published_time":"2025-02-04T19:36:22+00:00","article_modified_time":"2025-02-27T00:56:41+00:00","og_image":[{"width":2000,"height":1000,"url":"http:\/\/blog.daed.com\/wp-content\/uploads\/2025\/02\/6.png","type":"image\/png"}],"author":"Daedalus","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Daedalus","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.daed.com\/?p=4719#article","isPartOf":{"@id":"https:\/\/blog.daed.com\/?p=4719"},"author":{"name":"Daedalus","@id":"https:\/\/blog.daed.com\/#\/schema\/person\/ffe3d55f759956aa85792c64b0d0f984"},"headline":"The History of Neural Networks","datePublished":"2025-02-04T19:36:22+00:00","dateModified":"2025-02-27T00:56:41+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.daed.com\/?p=4719"},"wordCount":1038,"commentCount":0,"publisher":{"@id":"https:\/\/blog.daed.com\/#organization"},"articleSection":["design thinking","Software Engineering"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.daed.com\/?p=4719#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.daed.com\/?p=4719","url":"https:\/\/blog.daed.com\/?p=4719","name":"The History of Neural Networks - daed.com","isPartOf":{"@id":"https:\/\/blog.daed.com\/#website"},"datePublished":"2025-02-04T19:36:22+00:00","dateModified":"2025-02-27T00:56:41+00:00","description":"Neural networks are machine learning algorithms inspired by biological neural circuits. The human brain is incredibly complex. Lucky for software engineers, neural networks in ML are less complex (at least slightly)!","breadcrumb":{"@id":"https:\/\/blog.daed.com\/?p=4719#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.daed.com\/?p=4719"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.daed.com\/?p=4719#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.daed.com\/"},{"@type":"ListItem","position":2,"name":"The History of Neural Networks"}]},{"@type":"WebSite","@id":"https:\/\/blog.daed.com\/#website","url":"https:\/\/blog.daed.com\/","name":"daed.com","description":"research, design, and engineering thinking","publisher":{"@id":"https:\/\/blog.daed.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.daed.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/blog.daed.com\/#organization","name":"daed.com","url":"https:\/\/blog.daed.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.daed.com\/#\/schema\/logo\/image\/","url":"https:\/\/blog.daed.com\/wp-content\/uploads\/2019\/10\/White_Daedalus.png","contentUrl":"https:\/\/blog.daed.com\/wp-content\/uploads\/2019\/10\/White_Daedalus.png","width":5249,"height":745,"caption":"daed.com"},"image":{"@id":"https:\/\/blog.daed.com\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/blog.daed.com\/#\/schema\/person\/ffe3d55f759956aa85792c64b0d0f984","name":"Daedalus","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.daed.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/8bf29d93ebf6e34ef10f067eac16bcb56fdf8a13ae323fdda0b07b25d15d6c61?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8bf29d93ebf6e34ef10f067eac16bcb56fdf8a13ae323fdda0b07b25d15d6c61?s=96&d=mm&r=g","caption":"Daedalus"},"url":"https:\/\/blog.daed.com\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/posts\/4719","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.daed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4719"}],"version-history":[{"count":4,"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/posts\/4719\/revisions"}],"predecessor-version":[{"id":4747,"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/posts\/4719\/revisions\/4747"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.daed.com\/index.php?rest_route=\/wp\/v2\/media\/4770"}],"wp:attachment":[{"href":"https:\/\/blog.daed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4719"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.daed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4719"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.daed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4719"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}