Deep Learning2020. 1. 3. 23:39

[머신러닝/딥러닝] 딥러닝으로 MNIST 98%이상

반응형

※ 김성훈 교수님의 [모두를 위한 딥러닝] 강의 정리

 - https://www.youtube.com/watch?reload=9&v=BS6O0zOGX4E&feature=youtu.be&list=PLlMkM4tgfjnLSOjrEJN31gZATbcj_MpUm&fbclid=IwAR07UnOxQEOxSKkH6bQ8PzYj2vDop_J0Pbzkg3IVQeQ_zTKcXdNOwaSf_k0

 - 참고자료 : Andrew Ng's ML class

  1) https://class.coursera.org/ml-003/lecture

  2) http://holehouse.org/mlclass/ (note)



1. Softmax classifier for MNIST : Accuracy 0.9035

 - (중요 코드)

# weights & bias for softmax classifier
  hypothesis = tf.matmul(X, W) + b
# define cost/loss & optimizer
  cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=hypothesis, labels=Y))
  optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

 

2. NN for MNIST : Accuracy 0.9455

 - (중요 코드) 3단으로 늘림, ReLU 사용

# weights & bias for nn layers
W1 = tf.Variable(tf.random_normal([784, 256]))
b1 = tf.Variable(tf.random_normal([256]))
L1 = tf.nn.relu(tf.matmul(X, W1) + b1)
 
W2 = tf.Variable(tf.random_normal([256, 256]))
b2 = tf.Variable(tf.random_normal([256]))
L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)
 
W3 = tf.Variable(tf.random_normal([256, 10]))
b3 = tf.Variable(tf.random_normal([10]))
hypothesis = tf.matmul(L2, W3) + b3
 
# define cost/loss & optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=hypothesis, labels=Y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

3. Xavier Initialization for MNIST (초기화를 잘 하기) : Accuracy 0.9779

 - (중요 코드) 초기값 initializer만 변경

# weights & bias for nn layers
# http://stackoverflow.com/questions/33640581/how-to-do-xavier-initialization-on-tensorflow
W1 = tf.get_variable("W1", shape=[784, 256],
initializer=tf.contrib.layers.xavier_initializer())
b1 = tf.Variable(tf.random_normal([256]))
L1 = tf.nn.relu(tf.matmul(X, W1) + b1)
 
W2 = tf.get_variable("W2", shape=[256, 256],
initializer=tf.contrib.layers.xavier_initializer())
b2 = tf.Variable(tf.random_normal([256]))
L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)
 
W3 = tf.get_variable("W3", shape=[256, 10],
initializer=tf.contrib.layers.xavier_initializer())
b3 = tf.Variable(tf.random_normal([10]))
hypothesis = tf.matmul(L2, W3) + b3

 

4. Deep NN for MNIST : Accuracy 0.9742 

 - (중요 코드) 5단으로 늘림, 더 넓힘. 깊고 넓게~ => but 결과는 실망 (아마도 overfitting)..

# weights & bias for nn layers
# http://stackoverflow.com/questions/33640581/how-to-do-xavier-initialization-on-tensorflow
W1 = tf.get_variable("W1", shape=[784, 512],
initializer=tf.contrib.layers.xavier_initializer())
b1 = tf.Variable(tf.random_normal([512]))
L1 = tf.nn.relu(tf.matmul(X, W1) + b1)
 
W2 = tf.get_variable("W2", shape=[512, 512],
initializer=tf.contrib.layers.xavier_initializer())
b2 = tf.Variable(tf.random_normal([512]))
L2 = tf.nn.relu(tf.matmul(L1, W2) + b2)
 
W3 = tf.get_variable("W3", shape=[512, 512],
initializer=tf.contrib.layers.xavier_initializer())
b3 = tf.Variable(tf.random_normal([512]))
L3 = tf.nn.relu(tf.matmul(L2, W3) + b3)
 
W4 = tf.get_variable("W4", shape=[512, 512],
initializer=tf.contrib.layers.xavier_initializer())
b4 = tf.Variable(tf.random_normal([512]))
L4 = tf.nn.relu(tf.matmul(L3, W4) + b4)
 
W5 = tf.get_variable("W5", shape=[512, 10],
initializer=tf.contrib.layers.xavier_initializer())
b5 = tf.Variable(tf.random_normal([10]))
hypothesis = tf.matmul(L4, W5) + b5
 
# define cost/loss & optimizer
cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
logits=hypothesis, labels=Y))
optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

5. Dropout for MNIST : Accuracy 0.9804 

 - (중요 코드) 1개의 layer 추가만으로 tensorflow에서 구현 가능. 통상 0.5~0.7로 training 후 1로 testing!

W1 = tf.get_variable("W1", shape=[784, 512], initializer=tf.contrib.layers.xavier_initializer())
b1 = tf.Variable(tf.random_normal([512]))
L1 = tf.nn.relu(tf.matmul(X, W1) + b1)
L1 = tf.nn.dropout(L1, keep_prob=keep_prob)

...

# train my model
for epoch in range(training_epochs):
avg_cost = 0
 
for i in range(total_batch):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
feed_dict = {X: batch_xs, Y: batch_ys, keep_prob: 0.7}
c, _ = sess.run([cost, optimizer], feed_dict=feed_dict)
avg_cost += c / total_batch
 
print('Epoch:', '%04d' % (epoch + 1), 'cost =', '{:.9f}'.format(avg_cost))
 
print('Learning Finished!')
 
# Test model and check accuracy
correct_prediction = tf.equal(tf.argmax(hypothesis, 1), tf.argmax(Y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print('Accuracy:', sess.run(accuracy, feed_dict={
X: mnist.test.images, Y: mnist.test.labels, keep_prob: 1}))

 

6. Optimizers

 - GradientDescentOptimizer 外 어떤 알고리즘이 더 좋은지 Simulation 가능

- http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html  

 

- Adam Optimizer를 처음 써보는 것을 추천 : 기존 코드에서 GradientDescentOptimizer -> AdamOptimizer로만 변경

#define cost/loss & optimizer
cost =tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=hypothesis, labels=Y))
optimizer =tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)

 

반응형
저작자표시 비영리 변경금지

'Deep Learning' 카테고리의 다른 글

[머신러닝/딥러닝] Convolutional Neural Network(CNN) 활용사례  (0) 2020.01.07
[머신러닝/딥러닝] Convolutional Neural Network(CNN)  (0) 2020.01.03
[머신러닝/딥러닝] Dropout & Ensemble  (0) 2020.01.03
[머신러닝/딥러닝] Weight 초기화 잘해보자  (0) 2020.01.02
[머신러닝/딥러닝] Sigmoid << ReLU  (0) 2019.12.30
Posted by CCIBOMB
Deep Learning2019. 12. 18. 22:28

[머신러닝/딥러닝] MNIST Dataset

반응형

※ 김성훈 교수님의 [모두를 위한 딥러닝] 강의 정리

 - https://www.youtube.com/watch?reload=9&v=BS6O0zOGX4E&feature=youtu.be&list=PLlMkM4tgfjnLSOjrEJN31gZATbcj_MpUm&fbclid=IwAR07UnOxQEOxSKkH6bQ8PzYj2vDop_J0Pbzkg3IVQeQ_zTKcXdNOwaSf_k0

 - 참고자료 : Andrew Ng's ML class

  1) https://class.coursera.org/ml-003/lecture

  2) http://holehouse.org/mlclass/ (note)

 

1. MNIST Dataset

 - 숫자 0~9까지의 손글씨 이미지의 집합

 - 학습데이터 60,000개, 테스트데이터 10,000개로 구성

 - 사이즈는 28 x 28, 이미지의 값은 0 또는 1 (흑 또는 백)

 - preprocessing, formatting이 모두 완료된 데이터셋

 - 다운로드 주소 : http://yann.lecun.com/exdb/mnist/

 

2. TensorFlow에서 MNIST Dataset 불러오기 using input_data.py

 - 다운로드 주소 : https://github.com/tensorflow/tensorflow/blob/r0.7/tensorflow/examples/tutorials/mnist/input_data.py

from tensorflow.examples.tutorials.mnist import input_data
 
# Check out https://www.tensorflow.org/get_started/mnist/beginners for
# more information about the mnist dataset
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
 
nb_classes = 10
 
# MNIST data image of shape 28 * 28 = 784
X = tf.placeholder(tf.float32, [None, 784])
# 0 - 9 digits recognition = 10 classes
Y = tf.placeholder(tf.float32, [None, nb_classes])
 
W = tf.Variable(tf.random_normal([784, nb_classes]))
b = tf.Variable(tf.random_normal([nb_classes]))

 

3. Softmax

# Hypothesis (using softmax)
hypothesis = tf.nn.softmax(tf.matmul(X, W) + b)
 
cost = tf.reduce_mean(-tf.reduce_sum(Y * tf.log(hypothesis), axis=1))
train = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cost)
 
# Test model
is_correct = tf.equal(tf.argmax(hypothesis, 1), tf.argmax(Y, 1))
# Calculate accuracy
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))

 

4. Training epoch/batch

 - epoch : 전체 training set을 1회 학습

 - batch : 1 epoch을 나누어서 실행하기 위한 사이즈

# parameters
num_epochs = 15
batch_size = 100
num_iterations = int(mnist.train.num_examples / batch_size)
 
with tf.Session() as sess:
# Initialize TensorFlow variables
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(num_epochs):
avg_cost = 0
 
for i in range(num_iterations):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)
_, cost_val = sess.run([train, cost], feed_dict={X: batch_xs, Y: batch_ys})
avg_cost += cost_val / num_iterations
 
print("Epoch: {:04d}, Cost: {:.9f}".format(epoch + 1, avg_cost))
 
print("Learning finished")

 

5. Report results on test dataset

# Test the model using test sets
print(
"Accuracy: ",
accuracy.eval(
session=sess, feed_dict={X: mnist.test.images, Y: mnist.test.labels}
),
)

 

6. Sample image show and prediction

# Get one and predict
r = random.randint(0, mnist.test.num_examples - 1)
print("Label: ", sess.run(tf.argmax(mnist.test.labels[r : r + 1], 1)))
print(
"Prediction: ",
sess.run(tf.argmax(hypothesis, 1), feed_dict={X: mnist.test.images[r : r + 1]}),
)
 
plt.imshow(
mnist.test.images[r : r + 1].reshape(28, 28),
cmap="Greys",
interpolation="nearest",
)
plt.show()

 

7. 전체 소스코드

import tensorflow as tf
import matplotlib.pyplot as plt  # matplotlib : 이미지를 다루는 라이브러리
import random  # random : W와 b에 임의 값을 주는 역할
 
tf.set_random_seed(777) # for reproducibility
 
from tensorflow.examples.tutorials.mnist import input_data
 
# Check out https://www.tensorflow.org/get_started/mnist/beginners for
# more information about the mnist dataset
mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)  # MNIST Dataset 다운로드
 
nb_classes = 10  # 10개 label(0~9) 분류
 
# MNIST data image of shape 28 * 28 = 784 (이미지 feature 784개)
X = tf.placeholder(tf.float32, [None, 784])
# 0 - 9 digits recognition = 10 classes (이미지와 매칭되어 있는 label(정답) 10종류)
Y = tf.placeholder(tf.float32, [None, nb_classes])
 
W = tf.Variable(tf.random_normal([784, nb_classes]))  # W, b : 랜덤으로 값 지정
b = tf.Variable(tf.random_normal([nb_classes]))
 
# Hypothesis (using softmax)
hypothesis = tf.nn.softmax(tf.matmul(X, W) + b)  # 가설함수 H(X)를 softmax에 대입
 
cost = tf.reduce_mean(-tf.reduce_sum(Y * tf.log(hypothesis), axis=1))  # Cross-Entropy를 이용한 cost
train = tf.train.GradientDescentOptimizer(learning_rate=0.1).minimize(cost)  # Gradient Descent Algorithm을 이용한 cost 최소화
 
# Test model : 정답(label)과 H(X)의 예상결과 비교
is_correct = tf.equal(tf.argmax(hypothesis, 1), tf.argmax(Y, 1))
# Calculate accuracy : 정확도 측정
accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
 
# parameters
num_epochs = 15  # 전체 이미지를 15회 반복 학습
batch_size = 100  # 학습 이미지 파일이 많아서 100개씩 가져옴
num_iterations = int(mnist.train.num_examples / batch_size)
 
with tf.Session() as sess:
# Initialize TensorFlow variables : TensorFlow 변수 초기화 및 세션 시작
sess.run(tf.global_variables_initializer())
# Training cycle
for epoch in range(num_epochs):
avg_cost = 0
 
for i in range(num_iterations):
batch_xs, batch_ys = mnist.train.next_batch(batch_size)  # Training 데이터를 X와 Y에 너헝줌
_, cost_val = sess.run([train, cost], feed_dict={X: batch_xs, Y: batch_ys})  # 학습과정의 cost 계산
avg_cost += cost_val / num_iterations
 
print("Epoch: {:04d}, Cost: {:.9f}".format(epoch + 1, avg_cost))  # 1회 학습당 cost 결과 출력
 
print("Learning finished")
 
# Test the model using test sets : Test 데이터를 이용하여 label(정답)과 비교. 정확도 계산
print(
"Accuracy: ",
accuracy.eval(
session=sess, feed_dict={X: mnist.test.images, Y: mnist.test.labels}
),
)
 
# Get one and predict
r = random.randint(0, mnist.test.num_examples - 1)  # Test 데이터에서 임의의 값을 뽑아서 예측
print("Label: ", sess.run(tf.argmax(mnist.test.labels[r : r + 1], 1)))
print(
"Prediction: ",
sess.run(tf.argmax(hypothesis, 1), feed_dict={X: mnist.test.images[r : r + 1]}),
)
 
plt.imshow(
mnist.test.images[r : r + 1].reshape(28, 28),
cmap="Greys",
interpolation="nearest",
)
plt.show()  # Test 데이터에서 뽑은 이미지 출력

 

반응형
저작자표시 비영리 변경금지

'Deep Learning' 카테고리의 다른 글

[머신러닝/딥러닝] XOR 문제 딥러닝으로 풀기  (0) 2019.12.23
[머신러닝/딥러닝] Tensor Manipulation  (0) 2019.12.18
[머신러닝/딥러닝] 딥러닝의 기본 개념  (0) 2019.12.18
[머신러닝/딥러닝] 팁 : Learning rate, Preprocessing, Overfitting  (0) 2019.12.12
[머신러닝/딥러닝] Softmax Classification 구현하기 by TensorfFlow  (0) 2019.12.11
Posted by CCIBOMB

티스토리툴바