Yiyu's Murmur

You should play like you have nothing to lose.

听到师弟自杀的消息，还在震惊和悲伤中。

A normal month: 9.5K used.

• Home: 4k, this is the fixed and almost half of the monthly expense
• Grocery: 1.8k, roughly 1k at costco and 500 at Weee. Reduce the trips to Costco may help both my wallet and my weight.
• Shopping: 1.5k, finally build a new PC, spend 700 on motherboard/CPU/memory. $140 on apple pencil and $150 on some gc are the other major contributors.
• Kids: $500, a new bike, a new booster seat, dancing class (2 months), and some random books
• Pet: $80, medicine and some treat
• Restaurant: $100, only ordered carryout once.
• Utilities: $200
• Car: $500. including $464 insurance

Next month forecast: huge… prepaid some utilities, amazon and costco gc

(Following what other people has done: https://engineerseekingfire.com/may-2020-expense-report/ )

May is the month with lease spending even after we moved to Seattle, and also a two year celebration of us to move to Seattle, Join FB, and my son’s 6 year birthday.

This month we only spend $7159, including $3900 mortgage, purely due to the COVID-19.

Breakdown of expenses

• Home
  • Mortgage: $3900, including insurance, this is usually the largest expense in each month, and this month contributes to 55% of total expense. Hope after Oct this year, when I officially with H1B status, I can do a refinance and reduce this number. Do not have strong incentive to pay off some of the loans.
  • Other: $230 on a weber grill, which brought lots of fun and leisure food time for the family
• Grocery: $1100+, 15%, majorly from Weee and group buy with neighbors: fruits and seafoods
• Restaurant and Fast food: $250, a few deliveries, usually spend $1k before COVID-19
• Shopping: $1300, majorly from Amazon, and a $400 Video Card from Newegg and a $100+ case and $50 mouse from AMZ. Due to WFH and old PCs at home (both are build/manufactured in 2012), some of the parts are dying, and also want a SOIA 🙂 setting at home.
• Kids: $75 dancing class, and a $50+ LEGO, that is all for May. most of the gifts for Harry are counted in Shopping or last month.
• Others
  • $50 gas
  • $70 utilities: internet and mobiles, other utilities are either bimonth, or have prepaid in the account
  • some recurrent spending: amazon music, Wechat Read, Disney Plus.

Summary

• $7.1k will be a local minimum (of not a globale one) for my family’s monthly expense, majorly due to the pandemic, there are 0 travel and almost 0 dining outside. Also we did not buy anything large this month.
• So $80k will be a minimum number for yearly needs, and $120k will be a safer number, thus $3M (based on 4% rule) or $4M (based on 3%) is what I need, given my current net worth, I will need …. Well I gave it up. LOL
• I am a pretty aggressive people in terms of stock investment (or actually speculation), so to have $120k passive income, probably actually need $1M investement in stock market. This year’s investment (STOCK+401K+IRA) return till today: -0.46%

There is still a long way to FIRE

Recently I saw a blogger (probably from Reddit), and the concept of FIRE (FINANCIAL INDEPENDENCE AND EARLY RETIREMENT), and looks like I pretty like that idea: most of us do not need (and cannot) reach financial free, but can be financial independence, after which, one can work more freedom or do not work at all.

Key point:

• with a saving that is equals to 25X yearly expense, and 4% withdraw rate (in other words return rate to keep the saving stable)

Something to keep in mind

• Increase Income: can reach the needed savings early
• Decrease Expense: reduce unnecessary expenses
• Investment: that is how you get passive income

Start from tracking my yearly expense now.

Install Ubuntu 18.04 LTS

The old server has been dead for a while. Finally I bought a new PSU (not sure whether it is the PSU’s problem), reinstalled everything, organized cables, now I have new a desktop. Later I reinstalled the OS too, previously using a Windows Server 2012 for some reason (failure to install any linux at that time due to the nividia driver problem).

Surprisingly, this time it is a pretty smooth process to install a Ubuntu 18.04 LTS (with a AMD GPU at the beginning), and then switch to a nVidia GPU (old 4GB GTX 970).

In case you have problem install Ubuntu with a GTX GPU, try the solution of ‘blacklist nouveau driver’ in here

Install nVidia Drivers

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-driver-440
sudo reboot
# also you can install in setting -> advanced drivers

Install CUDA

If you want to delete old CUDA installed, try sudo apt-get –purge remove cuda.

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

Set CUDA ENV (not sure whether needed)

export PATH=$PATH:/usr/local/cuda-10.2/bin
export CUDADIR=/usr/local/cuda-10.2
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.2/lib64

Test Cuda

mkdir cuda-testing
cd cuda-testing/
cp -a /usr/local/cuda-10.2/samples samples-10.2
cd samples-10.2
make -j 4 # (add -k to skip errors)
~/cuda-testing/samples-10.2/bin/x86_64/linux/release$ ./nbody

Error

Cuda 10.2+Ubuntu 18.04 might give you error message of

'make: Target 'all' not remade because of errors.'

cudaNvSci.h:14:10: fatal error: nvscibuf.h: No such file or directory

 #include 

          ^~~~

compilation terminated.

Makefile:394: recipe for target 'cudaNvSci.o' failed

It is a temporary new feature issue according to https://github.com/NVIDIA/cuda-samples/issues/22#issuecomment-562105202

Install CuDNN

Download CuDNN lib for corresponding CUDA version from here

sudo dpkg -i  libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.2_amd64.deb

Test CuDNN Installation

cp -r /usr/src/cudnn_samples_v7/ ~/cuda-testing/cudnn_samples_v7/
~/cuda-testing/cudnn_samples_v7/mnistCUDNN$ make clean && make
./mnistCUDNN

Install Pytorch through Anaconda

sh ./Anaconda3-2020.02-Linux-x86_64.sh
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

#test
python -c 'import torch;print(torch.cuda.is_available())'
True

Install Tensorflow and Keras

from here

pip install tensorflow-gpu
pip install keras

conda install tensorflow-gpu keras

#test 
python -c 'from keras import backend as K;K.tensorflow_backend._get_available_gpus()'

from keras import backend as K 
K.tensorflow_backend._get_available_gpus()    

Install MXNet

pip install mxnet-cu102 d2lzh

Done

Reference

• Cuda Downloads: https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal
• Install Multiple Vertsion of Cuda and Test Cude Installation: https://www.pugetsystems.com/labs/hpc/How-To-Install-CUDA-10-together-with-9-2-on-Ubuntu-18-04-with-support-for-NVIDIA-20XX-Turing-GPUs-1236/
• https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html
• Anaconda: https://gist.github.com/kylemcdonald/3ae0b88a1bf91afc00ba441fe6823a17

DBN

• It is a Unsupervised Probabilistic generative graphical model to learn P(X), while LeNet/AlexNet and so on are discriminative models that focus on P(Y|X).
• The top two layers of the DBN form an undirected bipartite graph called Restricted Boltzmann Machine
• The lower layers form a directed sigmoid belief network
• DBN can be formed by “stacking” RBMs. Later Autoencoder is used instead.
• Greedy, layer-by-layer learning
• Optionally fine-tuned with gradient descent and backpropagation.

RBM

• RBMs are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph
• Architecture: RBM has an input layer (also referred to as the visible layer) and one single hidden layer and the connections among the neurons are restricted. So RBM looks like a MLP connection between two layers

Reference

• Hinton, Geoffrey E., Simon Osindero, and Yee-Whye Teh. “A fast learning algorithm for deep belief nets.” Neural computation 18.7 (2006): 1527-1554.
• Salakhutdinov, Ruslan, and Hugo Larochelle. “Efficient learning of deep Boltzmann machines.” Proceedings of the thirteenth international conference on artificial intelligence and statistics. 2010.
• Deep Architecture slides by Barnabas Poczos
• Deep Learning — Deep Belief Network (DBN)
• 2007 NIPS Tutorial on: Deep Belief Nets Geoffrey Hinton
• Reconstruction Of Images Using RBM

• Machine Learning/Deep Learning (This year I need to go deeper)
  • Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville
    • more theory and fundamental knowledge
  • 动手学深度学习 (Dive into Deep Learning) by Aston Zhang and Zachary C. Lipton and Mu Li and Alexander J. Smola
    • Focus on code and implementation, both MXNet and Pytorch
  • 机器学习 by 周志华
    • refresh knowledge of ML
  • 统计学习方法 by 李航
    • Did not realize that this Chinese book affected lots of ML researchers
  • 终极算法 (The Master Algorithm) by Pedro Domingos
    • Still do not know what this book is talking about, whether it is more of history or future
• (ML) System
  • Site Reliability Engineering
    • How to launch something with confident and easy to maintain
  • Designing Data-Intensive Applications
    • Might be the best book for data system design
  • 微服务设计 (Building Microservices: Designing Fine-Grained Systems) by Sam Newman
    • Always curious of what is the ideal of micro-services
• Productivity / Business & Economics / Time Management
  • Getting Things Done (1 to 3) by David Allen
    • Need to read this every year

Simply a collection of classic computer vision papers

Image Classification

• Gradient-based learning applied to document recognition, LeNet-5, Proceedings of the IEEE 1998, pdf
• ImageNet Classification with Deep Convolutional Neural Networks, AlexNet, NIPS 2012, pdf, slides
• Visualizing and Understanding Convolutional Networks, ZFNet, ECCV 2014, pdf
• Network In Network, NiN, ICLR 2014, pdf
• Very Deep Convolutional Networks for Large-Scale Image Recognition, VGG, pdf
• Going Deeper with Convolutions, Inception, CVPR 2015, pdf
• Deep Residual Learning for Image Recognition, ResNet, CVPR 2016, pdf
• Wide Residual Networks, BMVC 2016 , pdf
• Rethinking the Inception Architecture for Computer Vision, Inception v3, CVPR, pdf
• Aggregated Residual Transformations for Deep Neural Networks, ResNext, pdf
• Densely Connected Convolutional Networks, DenseNet, CVPR 2017, pdf
• Squeeze-and-Excitation Networks, SENet, CVPR 2018, pdf
• Residual Attention Network for Image Classification, CVPR 2018, pdf

Video Classification

• A Closer Look at Spatiotemporal Convolutions for Action Recognition, R(2+1)D, CVPR 2018, pdf
• Video Classification with Channel-Separated Convolutional Networks, CVPR 2019, pdf
• Large-scale weakly-supervised pre-training for video action recognition, pdf

MISC Resources

• Computer Vision Foundation open access, including CVPR and ICCV

Started to seriously read some NLP/CV/MulitModal publications, try to focus on fancy terms such as attention, fusion that are more frequently mentioned everywhere.

• An Attentive Survey of Attention Models from Linkedin AI 2019

‘Attention Model incorporates this notion of relevance by allowing the model to dynamically pay attention to only certain parts of the input that help in performing the task at hand effectively’

• Attention is all you need from Google, 2019

Transformer is proposed in this paper.