1. Redirection of standard output `>`

The default standard output is the screen.
> is output redirection symbol and syntax is:
$ command > output.file.name
https://bash.cyberciti.biz/guide/Standard_output

2. Redirected output `>>`

Appending the output to the same file using >> operator.

3. Pipes `|`

https://bash.cyberciti.biz/guide/Chapter_7:_Pipes_and_Filters

4. cat

Displaying The Contents of Files

$ cat filename

$ cat file > newfile

Use a pipe to filter data:

$ cat file | less

Concatenate files

$ cat file1 file2

$ cat file1 file2 > newcombinedfile

Create new text files

note that if a file already exists, it will be overwritten.

$ cat > filename

append the output to the same file using >> operator:

$ cat >> filename

Copy file

$ cat oldfile > newfile

https://www.cyberciti.biz/faq/howto-use-cat-command-in-unix-linux-shell-script/

5. grep & awk

grep 行处理 awk 列处理

cat jupyter_notebook_config.json | grep password | awk -F '"' '{print $4}'

PWD=$(cat jupyter_notebook_config.json | grep password | awk -F '"' '{print $4}')

6. chmod

$ chmod +x tarbackup.sh

$ ./tarbackup.sh

7. ps

ps -aux 显示所有包含其他使用者的行程

ps -aux | grep ssh 与grep结合，查找特定进程

ps -ef | grep PID , 找到其父进程，然后 kill -9 父进程ID

kill -9 pid

8. unzip

sudo apt-get install unzip

unzip file.zip -d destination_folde

unzip

9. tar

tar -czvf ***.tar.gz directory

tar -xzvf ***.tar.gz

https://www.cnblogs.com/52linux/archive/2012/03/04/2379738.html

10. windows ‘\r\n’ to linux ‘\n’

cat -A filename

vim filename && set ff=unix

dos2unix tool

11. rsync

upload: rsync -avh --progress --delete --exclude=analysis/ src_dir/ host:dest_dir

download: rsync -avh --progress host:dest_dir ./

-a, --archive     # 归档模式，表示以递归方式传输文件，并保持所有文件属性，等价于 -rlptgoD
-r, --recursive   # 对子目录以递归模式处理
-l, --links       # 保持符号链接文件
-H, --hard-links  # 保持硬链接文件
-p, --perms       # 保持文件权限
-t, --times       # 保持文件时间信息
-g, --group       # 保持文件属组信息
-o, --owner       # 保持文件属主信息 (super-user only)
-D                # 保持设备文件和特殊文件 (super-user only)

-v, --verbose     # 详细输出模式
-h, --human-readable # 输出文件大小使用易读的单位（如，K，M等）


--exclude=PATTERN # 指定排除一个不需要传输的文件匹配模式 -exclude="*.iso", --exclude={'file1.txt','dir1/*'} 
--exclude-from    # 指定一个本地文件，里面是需要排除的文件模式，每个模式一行。


--delete 
# 删除那些接收端还有而发送端已经不存在的文件。注意：如果--exclude传输排除的文件，--delete也不会删除接受端的这个文件。如果要删除，需要--delete-excluded指定。例如：rsync --delete exclude='.git' 接受端不会因--delete而删除发射端排除在外没同步的.git文件，即保留接受端原有的.git文件; 除非指定--delete-excluded='.git'，如rsync --delete exclude='.git' --delete-excluded='.git' 。



--progress        # 在传输时显示传输过程

-n参数或--dry-run参数模拟将要执行的操作，而并不真的执行。配合-v参数使用，可以看到哪些内容会被同步过去。

http://www.ruanyifeng.com/blog/2020/08/rsync.html

https://download.samba.org/pub/rsync/rsync.html

https://rsync.samba.org/how-rsync-works.html

12. SSH

SSH远程执行命令

远程执行命令：

ssh lsu1@xxx.xxx.xxx.xxx "df -h"
ssh lsu1@xxx.xxx.xxx.xxx "df -h; ls"

远程执行本地脚本：

ssh lsu1@xxx.xxx.xxx.xxx < test.sh
ssh lsu1@xxx.xxx.xxx.xxx 'bash -s' < test.sh helloworld # 带参数

执行远程服务器上脚本：

ssh lsu1@xxx.xxx.xxx.xxx "/home/lsu1/test.sh" # 需要绝对路径

无密钥登陆

SSH Client : 192.168.0.12

SSH Remote Host : 192.168.0.11

Create Authentication SSH-Kegen Keys on – 192.168.0.12

ssh-keygen -t rsa

ssh-keygen -t rsa -b 4096 -m pem

Create .ssh Directory on – 192.168.0.11

ssh 192.168.0.11 mkdir -p .ssh

Upload Generated Public Keys to – 192.168.0.11

cat .ssh/id_rsa.pub | ssh 192.168.0.11 'cat >> .ssh/authorized_keys'

best way: ssh-copy-id -i ~/.ssh/id_rsa.pub lsu1@139.224.58.222

Set Permissions on – 192.168.0.11

ssh 192.168.0.11 "chmod 700 .ssh; chmod 640 .ssh/authorized_keys"

Set StrictModes

sudo vim /etc/ssh/sshd_config ===> StrictModes no

sudo service ssh restart

https://www.tecmint.com/ssh-passwordless-login-using-ssh-keygen-in-5-easy-steps

Auto disconnect

/etc/ssh/sshd_config

ClientAliveInterval 60
ClientAliveCountMax 3

restart sshd：

Linux: service sshd restart

MAC: service: command not found

MAC: brew services start sshd

13. pip

cd ~ && mkdir .pip && vim .pip/pip.conf

[global]
index-url = https://pypi.doubanio.com/simple/
timeout = 1000
[install]
use-mirrors = true
mirrors = https://pypi.doubanio.com//

14. 设置环境变量

Shell临时设置, 终端关闭后失效

export PATH=/cygdrive/c/Users/YCKJ2939/anaconda3:$PATH

~/.bashrc永久设置，只对本用户可见

echo 'export PATH=/cygdrive/c/Users/YCKJ2939/anaconda3:$PATH' >> ~/.bashrc

source ~./bashrc

/etc/profile永久设置，对所有用户可见

How To Read and Set Environmental and Shell Variables on a Linux VPS

15. lsof

lsof -i

16. siege

An HTTP/HTTPS stress tester

siege http://localhost:8000/?q=pants -c10 -t10s

17. supervisor

Supervisor process control system for UNIX

18. brew

MAC install brew

19. iTerm2 rzsz

https://juejin.cn/post/6844904176707698695

20. lsof

端口号占用情况lsof -i:port

22. hostname

hostname -i

23. kill

ps -ef|grep nginx|grep -v grep|cut -c 9-15|xargs kill -9

https://www.joshua317.com/article/36

24. iTerm2 + powerline + oh-my-zsh

25. iconv

iconv -c -f gbk -t utf8 file_name -o new_file_name

26. /dev

lsu1@lsu1:~# lsblk
NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    252:0    0  40G  0 disk
└─vda1 252:1    0  40G  0 part /

lsu1@lsu1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            975M     0  975M   0% /dev
tmpfs           200M  6.5M  193M   4% /run
/dev/vda1        40G   28G   11G  73% /
tmpfs           997M     0  997M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           997M     0  997M   0% /sys/fs/cgroup
overlay          40G   28G   11G  73% /var/lib/docker/overlay2/2263075e71fe3ecd347c2a33571546c607048e740de7da2eac72eec365c6b429/merged
tmpfs           200M     0  200M   0% /run/user/0

lsblk查看可用可设备，df -h块设备vda1块设备挂在到目录/，mount与umount挂在和卸载挂载点。

mount /dev/nvme3n1 /data 挂载nvme3n1硬盘到/data目录

echo "/dev/nvme3n1 /data ext4 defaults 0 0" >> /etc/fstab # 保证重启生效

ln

ln -s /home/data /data:创建目录/home/data的软连接/data
rm -rf /data：删除软连接，一定不能rm -rf /data/linux默认会不全/，这样删除会把/home/data/目标目录下的内容全部删除！！！

Read more:

≪ Scala Debugging Spark Application ≫

1. Redirection of standard output >

2. Redirected output >>

3. Pipes |