Yiyu's Murmur

DBN

• It is a Unsupervised Probabilistic generative graphical model to learn P(X), while LeNet/AlexNet and so on are discriminative models that focus on P(Y|X).
• The top two layers of the DBN form an undirected bipartite graph called Restricted Boltzmann Machine
• The lower layers form a directed sigmoid belief network
• DBN can be formed by “stacking” RBMs. Later Autoencoder is used instead.
• Greedy, layer-by-layer learning
• Optionally fine-tuned with gradient descent and backpropagation.

RBM

• RBMs are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph
• Architecture: RBM has an input layer (also referred to as the visible layer) and one single hidden layer and the connections among the neurons are restricted. So RBM looks like a MLP connection between two layers

Reference

• Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. “A fast learning algorithm for deep belief nets.” Neural computation 18.7 (2006): 1527-1554.
• Salakhutdinov, Ruslan, and Hugo Larochelle. “Efficient learning of deep Boltzmann machines.” Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010.
• Deep Architecture slides by Barnabas Poczos
• Deep Learning — Deep Belief Network (DBN)
• 2007 NIPS Tutorial on: Deep Belief Nets Geoffrey Hinton
• Reconstruction Of Images Using RBM

• Machine Learning/Deep Learning (This year I need to go deeper)
  • Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville
    • more theory and fundamental knowledge
  • 动手学深度学习 (Dive into Deep Learning) by Aston Zhang and Zachary C. Lipton and Mu Li and Alexander J. Smola
    • Focus on code and implementation, both MXNet and Pytorch
  • 机器学习 by 周志华
    • refresh knowledge of ML
  • 统计学习方法 by 李航
    • Did not realize that this Chinese book affected lots of ML researchers
  • 终极算法 (The Master Algorithm) by Pedro Domingos
    • Still do not know what this book is talking about, whether it is more of history or future
• (ML) System
  • Site Reliability Engineering
    • How to launch something with confident and easy to maintain
  • Designing Data-Intensive Applications
    • Might be the best book for data system design
  • 微服务设计 (Building Microservices: Designing Fine-Grained Systems) by Sam Newman
    • Always curious of what is the ideal of micro-services
• Productivity / Business & Economics / Time Management
  • Getting Things Done (1 to 3) by David Allen
    • Need to read this every year

Simply a collection of classic computer vision papers

Image Classification

• Gradient-based learning applied to document recognition, LeNet-5, Proceedings of the IEEE 1998, pdf
• ImageNet Classification with Deep Convolutional Neural Networks, AlexNet, NIPS 2012, pdf, slides
• Visualizing and Understanding Convolutional Networks, ZFNet, ECCV 2014, pdf
• Network In Network, NiN, ICLR 2014, pdf
• Very Deep Convolutional Networks for Large-Scale Image Recognition, VGG, pdf
• Going Deeper with Convolutions, Inception, CVPR 2015, pdf
• Deep Residual Learning for Image Recognition, ResNet, CVPR 2016, pdf
• Wide Residual Networks, BMVC 2016 , pdf
• Rethinking the Inception Architecture for Computer Vision, Inception v3, CVPR, pdf
• Aggregated Residual Transformations for Deep Neural Networks, ResNext, pdf
• Densely Connected Convolutional Networks, DenseNet, CVPR 2017, pdf
• Squeeze-and-Excitation Networks, SENet, CVPR 2018, pdf
• Residual Attention Network for Image Classification, CVPR 2018, pdf

Video Classification

• A Closer Look at Spatiotemporal Convolutions for Action Recognition, R(2+1)D, CVPR 2018, pdf
• Video Classification with Channel-Separated Convolutional Networks, CVPR 2019, pdf
• Large-scale weakly-supervised pre-training for video action recognition, pdf

MISC Resources

• Computer Vision Foundation open access, including CVPR and ICCV

Started to seriously read some NLP/CV/MulitModal publications, try to focus on fancy terms such as attention, fusion that are more frequently mentioned everywhere.

• An Attentive Survey of Attention Models from Linkedin AI 2019

‘Attention Model incorporates this notion of relevance by allowing the model to dynamically pay attention to only certain parts of the input that help in performing the task at hand effectively’

• Attention is all you need from Google, 2019

Transformer is proposed in this paper.

Start to plan to read 52 26 books per year. Considering that I have already finished 14 book in Wechat Reading since June, so it should be a P75 target. 1/3 of the books will be novel and whatever for recreational purpose. 1/3 for NON-tech but personal improvement such as GTD/time management/finance. The last 1/3 for Tech like new language (seems I am no longer interested in trying new language), ML/DL (major focus), software engineering and so on.

For Tech:

1. Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville
2. Model-Based Machine Learning by Bishop (not sure whether I want to read PRML again, so just start the new book)
3. The Master Algorithm by Pedro Domingos (bought a Chinese version in Wechat Reading…)
4. Some Caffe/PyTorch/TF coding book

For Personal Growth

1. Getting Things Done I (Bought in Wechat Reading and also Douban), but never finished. 50%
2. Deep Work
3. Outliers: The Story of Success
4. A Message to Garcia: And Other Essential Writings on Success (I cannot believe I finished reading this one)
5. One Up On Wall Street by Peter Lynch and John Rothchild
6. So Good They CAN’t Ignore You
7. Pomodoro Technique Illustrated by Staffan Noteberg

For Amusement

1. Origin by Dan Brown

 

Did not realize that VPS (Virtual Private Server) has been made so easy and so inexpensive. Last week, when reading the book “Deep Work“, suddenly realized that I am out of school, and again be free to write my ideas publicly. However, the static pages hosted github.io using markdown is kind of less efficient and hard (easy to forget) to include meta information. An inexpensive VPS (with static IP) can be a better choice. Find a good VPS has been an task in my list… until I remembered it, and googled, and saw the amazon lightsail.

A really lifesaver, log in with amazon count, a few clicks, here it is. I have a new wordpress blog with static IP. Find the user name and password for the wordpress takes some time: user name is user, password is in the VPS: login to your vps via SSH, and the pwd is stored in the file called bitnami_application_password.

Here we go, changed to some simple theme, write some intro (hesitate to put my resume here), and wrote the first post. Done.

Happy Fourth of July!