leelearns

Foundations of Neural Networks

In my last post, I was asking the question, "What does open-weight mean?". This led to me learning about neural networks. I'm going to try building a neural network myself right now, and I'll share what I find interesting throughout this adventure.

Imagine you are trying to decide whether or not now is a good time to go on a hike. What questions would you be asking yourself? For me, it would be things like, "what's the weather like?", "how long is it?", "how much time would it take?".

Some of those questions are more important than others. Even if I have enough time for the hike, I'm not gonna go if there is a tornado nearby. Therefore, the weather carries a heavier weight than the duration. In neural networks, weights are expressed as numbers, and they adjust. All of those adjustable numbers are often called parameters. The word configuration kinda makes more sense to me.

temperature = 79 # degrees f
weight = 0.8     # it 80% important. take 80% of the input.
signal = temperature * weight
print(signal)

day_of_week = 2
weight = 0.0  # ignore this import entirely
signal = day_of_week * weight
print(signal) # 0.0 weight makes everything 0

chance_of_tornado = 99
weight = -1.0
signal = chance_of_tornado * weight
print(signal) # negative weight goes against the answer

A weighted sum is the sum of all weights.

temperature = 79
day_of_week = 2
chance_of_tornado = 99

temp_weight = 0.8    
day_weight = 0.0  
tornado_weight = -1.0

weighted_sum = (temperature * temp_weight) + (day_of_week * day_weight) + (chance_of_tornado * tornado_weight)
print(weighted_sum)

A bias is kind of a hack. It's a number that doesn't multiply anything, and it used to boost (or sink) importance.

temperature = 79
day_of_week = 2
chance_of_tornado = 99

temp_weight = 0.8    
day_weight = 0.0  
tornado_weight = -1.0

weighted_sum = (temperature * temp_weight) + (day_of_week * day_weight) + (chance_of_tornado * tornado_weight)
print(weighted_sum)

bias = -2.0
z = weighted_sum + bias
print(z)

The most repeated operation a neural network does is: sum(weights * inputs) + bias

In practice, biases are sometimes fused into the weights so the extra add operation doesn't need to be done.

But this number could be anything... 10.5, -68748, 2,001,128. How do these help answer the question, should I go for a hike?

The sigmoid function takes in a number and sqashes it into a number between 0 and 1.

import numpy as np

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

print(sigmoid(10.5))
print(sigmoid(0))
print(sigmoid(-10.5))
print(sigmoid(2_092_010))

Sigmoid is an example of an activation function, which are formulas that are essential for ... ?

In a neural network, a neuron consists of:

A neural network consists of neurons organized into layers. Every network has at least two layers.