马哥Linux培训课堂笔记(3)

2017-11-14|Categories: Magedu-training|

以下内容主要按照在课堂讲述的顺序记录,有适当整理和延伸,便于回顾复习。

重要知识点

Linux文件系统

ext家族文件系统是Linux原生支持的、最正统的文件系统,它大致包含了以下信息:

  • data block:存放文件内容
  • inode table:存放文件元数据
  • superblock:存放文件系统元数据
  • 其它……

data block

  • 每个block只能存放一个文件的数据
  • 如果文件比单个block大,就占用多个block
  • 如果文件比单个block小,block的剩余空间就无法再使用
  • block的大小只能通过格式化来改变

inode table

每一个文件有一个inode,它至少保存了以下元数据:

注意:

  • inode不保存文件名,文件名保存在目录中。这样做的原因是为了能够创建文件的硬链接。
  • 间接记录指针是按需创建的,假设每个block大小为4096 bytes,有一个文件需要占用12个block,那就只会使用12个直接记录指针;当文件增大,需要占用更多的block时,才需要创建间接记录指针。
  • 当inode耗尽,即使磁盘上还有空的block,也无法创建新文件。
  • inode删除后,会被重新分配给新文件。

superblock

superblock是文件系统最重要的部分,它记录了以下信息:

CentOS 6使用ext4文件系统,默认:

  • 每个block大小为4096 bytes。
  • 每个inode大小为256 bytes。

CentOS 7使用xfs文件系统,默认:

  • 每个block大小为4096 bytes。
  • 每个inode大小为512 bytes。

硬链接 vs 软链接

硬链接就是同一个inode有多个文件名,这多个不同名称的文件本质上是同一个文件。软链接就是保存了指向另一个文件的路径,本质上是不同的文件。

硬链接和软链接的区别如下:

区别 硬链接 软链接 备注
跨文件系统创建 inode编号只有在同一个文件系统内才是独一无二的,多个文件系统会出现相同的inode编号
针对目录创建 Why are hard links not allowed for directories?
针对不存在的文件创建 不存在的文件没有inode编号
链接数增加 软链接是一个完全不同的文件
指向文件被删除后仍然正常工作 删除硬链接只会减少链接数,只要链接数大于0,文件inode编号就不会被删除

因为软链接更灵活,使用的频率会更高:

[liyang@centos7 testing]$ mkdir -p dir1/dir2/dir3
[liyang@centos7 testing]$ tree
.
├── dir1
│   └── dir2
│       └── dir3
├── test-pipe
└── test.txt

3 directories, 2 files

[liyang@centos7 testing]$ cd dir1/dir2/dir3/
# 用相对路径创建软链接,不是相对于当前工作目录,而是相对于将要创建的软链接的路径
[liyang@centos7 dir3]$ ln -s ../../../test-pipe test-pipe-linked
[liyang@centos7 dir3]$ ll test-pipe-linked
lrwxrwxrwx 1 liyang liyang 18 Nov 14 20:44 test-pipe-linked -> ../../../test-pipe

[liyang@centos7 dir3]$ cd ../../../
[liyang@centos7 testing]$ tree
.
├── dir1
│   └── dir2
│       └── dir3
│           └── test-pipe-linked -> ../../../test-pipe
├── test-pipe
└── test.txt

3 directories, 3 files

readlink – 显示软链接指向的真实文件

只对软链接有效。

print resolved symbolic links or canonical file names

from man readlink

[liyang@centos7 ~]$ readlink /bin
usr/bin

I/O重定向

标准I/O设备

  • stdin标准输入:键盘
  • stdout标准输出:屏幕
  • stderr标准错误:屏幕

文件描述符

Linux系统上每个打开的文件都有一个编号,叫做文件描述符(File Descriptor = FD)。Linux给标准I/O设备指定的fd如下:

  • fd 0 = stdin
  • fd 1 = stdout
  • fd 2 = stderr

通过I/O重定向技术,可以改变默认的I/O方向:

[liyang@centos7 ~]$ nofile
bash: nofile: command not found...
# 标准错误重定向到设备
[liyang@centos7 ~]$ nofile 2> /dev/null
# 标准错误重定向到文件
[liyang@centos7 ~]$ nofile 2> test.txt
[liyang@centos7 ~]$ cat test.txt
bash: nofile: command not found...

[liyang@centos7 ~]$ ll
total 4.0K
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Desktop
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Documents
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Downloads
# 省略多行……
# 标准输出重定向到文件
[liyang@centos7 ~]$ ll > ll-test.txt
[liyang@centos7 ~]$ cat ll-test.txt
total 4.0K
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Desktop
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Documents
drwxr-xr-x. 2 liyang liyang  6 Nov  7 16:27 Downloads
# 省略多行……

单行重定向

单行重定向的特点:

  • 每按一下回车键就写入一次。
  • 写入时不会展开命令或变量。
[liyang@centos7 ~]$ cat > single-line-redirection.txt
hello `whoami`
current path is $PATH

[root@centos7 ~] cat ~liyang/single-line-redirection.txt
hello `whoami`
current path is $PATH

多行重定向

多行重定向的特点:

  • 遇到结束token才统一写入。
  • 写入时会展开命令或变量。
[liyang@centos7 ~]$ cat > multi-line-redirection.txt <<eof
> hello `whoami`
> current path is $PATH
> eof

[root@centos7 ~] cat ~liyang/multi-line-redirection.txt
hello liyang
current path is /usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/liyang/.local/bin:/home/liyang/bin

禁止>重定向覆盖

重定向符号>可以轻易清空文件内容,且没有提示,比较危险,可以将其禁用:

[liyang@centos7 testing]$ cat test.txt
something for testing.

# 禁止`>`覆盖
[liyang@centos7 testing]$ set -C
# `>`覆盖失败
[liyang@centos7 testing]$ echo "I will overwrite the file." > test.txt
-bash: test.txt: cannot overwrite existing file

# 允许`>`覆盖
[liyang@centos7 testing]$ set +C
[liyang@centos7 testing]$ echo "I will overwrite the file." > test.txt
[liyang@centos7 testing]$ cat test.txt
I will overwrite the file.

[liyang@centos7 testing]$ set -C
# 用`>|`强制覆盖
[liyang@centos7 testing]$ echo "I will overwrite the file with 'set -C'." >| test.txt
[liyang@centos7 testing]$ cat test.txt
I will overwrite the file with 'set -C'.

另外,重定向符号>>>修改文件内容不会改变文件的时间戳

管道

管道是一种非常方便的进程间通信技术,可以方便的把前一个命令的标准输出转化为后一个命令的标准输入:

[liyang@centos7 ~]$ ll | tee ll.out | tr 'a-z' 'A-Z'
TOTAL 16K
DRWXR-XR-X. 2 LIYANG LIYANG   6 NOV  7 16:27 DESKTOP
DRWXR-XR-X. 2 LIYANG LIYANG   6 NOV  7 16:27 DOCUMENTS
DRWXR-XR-X. 2 LIYANG LIYANG   6 NOV  7 16:27 DOWNLOADS
-RW-RW-R--  1 LIYANG LIYANG 591 NOV 14 16:38 LL-TEST.TXT
-RW-RW-R--  1 LIYANG LIYANG 120 NOV 14 16:49 MULTI-LINE-REDIRECTION.TXT

[liyang@centos7 ~]$ cat ll.out
total 16K
drwxr-xr-x. 2 liyang liyang   6 Nov  7 16:27 Desktop
drwxr-xr-x. 2 liyang liyang   6 Nov  7 16:27 Documents
drwxr-xr-x. 2 liyang liyang   6 Nov  7 16:27 Downloads
-rw-rw-r--  1 liyang liyang 591 Nov 14 16:38 ll-test.txt
-rw-rw-r--  1 liyang liyang 120 Nov 14 16:49 multi-line-redirection.txt

管道默认只会处理标准输出,不会处理标准错误,但可以把标准错误转换为标准输出:

# 默认不处理标准错误
[liyang@centos7 testing]$ ll nofile | tr 'a-z' 'A-Z'
ls: cannot access nofile: No such file or directory

# 用`2>&1`把标准错误重定向为标准输出
[liyang@centos7 testing]$ ll nofile 2>&1 | tr 'a-z' 'A-Z'
LS: CANNOT ACCESS NOFILE: NO SUCH FILE OR DIRECTORY

# 用`|&`把标准错误强制处理为标准输出
[liyang@centos7 testing]$ ll nofile |& tr 'a-z' 'A-Z'
LS: CANNOT ACCESS NOFILE: NO SUCH FILE OR DIRECTORY

涉及的命令

hexdump

# 生成一个带有Unix换行的普通文件
[liyang@centos7 ~]$ echo -e 'I am an Unix file.\nThere is a new line.' > unix.txt

# 转换成一个新的带有Windows换行的文件
[liyang@centos7 ~]$ unix2dos -n unix.txt win.txt
unix2dos: converting file unix.txt to file win.txt in DOS format ...

# 查看两个文件的类型
[liyang@centos7 ~]$ file unix.txt win.txt
unix.txt: ASCII text
win.txt:  ASCII text, with CRLF line terminators

# 用十六进制查看Windows换行文件,换行符是`\r\n`
[liyang@centos7 ~]$ hexdump -c win.txt
0000000   I       a   m       a   n       U   n   i   x       f   i   l
0000010   e   .  \r  \n   T   h   e   r   e       i   s       a       n
0000020   e   w       l   i   n   e   .  \r  \n
000002a

# 用十六进制查看Unix换行文件,换行符是`\n`
[liyang@centos7 ~]$ hexdump -c unix.txt
0000000   I       a   m       a   n       U   n   i   x       f   i   l
0000010   e   .  \n   T   h   e   r   e       i   s       a       n   e
0000020   w       l   i   n   e   .  \n
0000028

# 因为多了两个回车符,win.txt要大2 bytes
[liyang@centos7 ~]$ ll win.txt unix.txt
-rw-rw-r-- 1 liyang liyang 40 Nov 14 17:13 unix.txt
-rw-rw-r-- 1 liyang liyang 42 Nov 14 17:14 win.txt

exec

本小节内容全部来自 https://www.computerhope.com/unix/bash/exec.htm

About exec

exec is a builtin command of the Bash shell. It allows you to execute a command that completely replaces the current process. The current shell process is destroyed, and entirely replaced by the command you specify.

exec is a critical function of any Unix-like operating system. Traditionally, the only way to create a new process in Unix is to fork it. The fork system call makes a copy of the forking program. The copy then uses exec to execute the child process in its memory space.

Syntax

exec [-c] [-l] [-a name] [command [arguments ...]] [redirection ...]

Description

exec is useful when you want to run a command, but you don't want a bash shell to be the parent process. When you exec a command, it replaces bash entirely — no new process is forked, no new PID is created, and all memory controlled by bash is destroyed and overwritten. This can be useful if, for instance, you want to give a user restricted access to a certain command. If the command exits because of an error, the user will not be returned to the privileged shell that executed it.

exec may also be used without any command, to redirect all output of the current shell to a file.

Examples

exec rbash

Replace the current bash shell with rbash, the restricted bash login shell. Because the original bash shell is destroyed, the user will not be returned to a privileged bash shell when rbash exits.

exec > output.txt

Redirect all output to the file output.txt for the current shell process. Redirections are a special case, and exec does not destroy the current shell process, but bash will no longer print output to the screen, writing it to the file instead. (This technique is much more useful in scripts — if the above command is executed in a script, all script output will be written to output.txt.)

exec 3< myinfile.txt

Open myinfile.txt for reading ("<") on file descriptor 3.

After running the above command, you can read a line of myinfile.txt by running the read command with the -u option:

read -u 3 mydata

Here, "-u 3" tells read to get its data from file descriptor 3, which refers to myinfile.txt. The contents are read, one line at a time, into the variable mydata. This would be useful if used as part of a while loop, for example.

Let's look at some more commands that open and close new file descriptors.

exec 4> out.txt

The above command opens out.txt for writing (">") on file descriptor 4.

exec 3<&-

Close ("&-") the open read descriptor ("<") number 3.

exec 4>&-

Close the open write descriptor (">") number 4.

exec 5<> myfile.txt

Open myfile.txt for reading and writing ("<>") as file descriptor 5.

exec 5<>&-

Close open read/write descriptor 5.

exec 6>> myappendfile.txt

Open myappendfile.txt for appending (">>") as file descriptor 6.

exec {myfd}> myfile.txt

Open myfile.txt for writing. A new file descriptor number, chosen automatically, is assigned to the variable myfd.

echo Text >&myfd

Echo the text "Text", and redirect the output to the file (in this case, myfile.txt) described by the write descriptor (">") whose number is obtained by dereferencing ("&") the variable named myfd.

在特定的file descriptor打开文件

[liyang@centos7 testing]$ echo 'something for testing.' > test.txt

[liyang@centos7 testing]$ cat test.txt
something for testing.

[liyang@centos7 testing]$ exec 5< test.txt

[liyang@centos7 testing]$ ll /proc/$$/fd
total 0
lrwx------ 1 liyang liyang 64 Nov 14 16:09 0 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 16:09 1 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 16:09 2 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 19:56 255 -> /dev/pts/1
lr-x------ 1 liyang liyang 64 Nov 14 16:09 5 -> /home/liyang/testing/test.txt

[liyang@centos7 testing]$ cat /proc/$$/fd/5
something for testing.

[liyang@centos7 testing]$ exec 5<&-

[liyang@centos7 testing]$ cat /proc/$$/fd/5
cat: /proc/4517/fd/5: No such file or directory

[liyang@centos7 testing]$ ll /proc/$$/fd
total 0
lrwx------ 1 liyang liyang 64 Nov 14 16:09 0 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 16:09 1 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 16:09 2 -> /dev/pts/1
lrwx------ 1 liyang liyang 64 Nov 14 19:56 255 -> /dev/pts/1

技巧

watch – 定期执行命令并输出结果

# 每隔一秒列出`/home/liyang`的文件
watch -n 1 ls -l ~liyang

删除文件

快速删除「大」文件

使用重定向技术清空文件内容再删除,效率比直接使用rm要高。

直接使用rm在虚拟机里删除5G文件,总时间0.928秒:

# 创建一个5G的文件
[liyang@centos7 ~]$ dd if=/dev/zero of=bigfile bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 24.3679 s, 215 MB/s

# 列出文件大小
[liyang@centos7 ~]$ ll bigfile
-rw-rw-r-- 1 liyang liyang 4.9G Nov 14 17:44 bigfile

# 用`time`命令统计任务执行时间
[liyang@centos7 ~]$ time /bin/rm bigfile

real    0m0.928s   # 任务执行总时间
user    0m0.000s
sys 0m0.163s

# 确认文件已删除
[liyang@centos7 ~]$ ll bigfile
ls: cannot access bigfile: No such file or directory

同样是虚拟机里的一个5G文件,先清空内容再删除,总时间0.185秒,只有直接删除的20%:

# 创建一个5G的文件
[liyang@centos7 ~]$ dd if=/dev/zero of=bigfile bs=1M count=5000
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 13.3214 s, 394 MB/s

# 列出文件大小
[liyang@centos7 ~]$ ll bigfile
-rw-rw-r-- 1 liyang liyang 4.9G Nov 14 17:45 bigfile

# 用`time`命令统计任务执行时间
[liyang@centos7 ~]$ time ( > bigfile && /bin/rm bigfile )

real    0m0.185s   # 任务执行总时间
user    0m0.000s
sys 0m0.177s

# 确认文件已删除
[liyang@centos7 ~]$ ll bigfile
ls: cannot access bigfile: No such file or directory

shred – 让删除的文件难以恢复

shred可以对要删除的文件随机写入内容,写入次数可以随意指定,使其难以恢复:

Overwrite the specified FILE(s) repeatedly, in order to make it harder for even very expensive hardware probing to recover the data.

from man shred

[liyang@centos7 testing]$ > test.txt

[liyang@centos7 testing]$ shred -zvun 9 test.txt
shred: test.txt: pass 1/10 (random)...
shred: test.txt: pass 2/10 (000000)...
shred: test.txt: pass 3/10 (492492)...
shred: test.txt: pass 4/10 (555555)...
shred: test.txt: pass 5/10 (random)...
shred: test.txt: pass 6/10 (ffffff)...
shred: test.txt: pass 7/10 (aaaaaa)...
shred: test.txt: pass 8/10 (249249)...
shred: test.txt: pass 9/10 (random)...
shred: test.txt: pass 10/10 (000000)...
shred: test.txt: removing
shred: test.txt: renamed to 00000000
shred: 00000000: renamed to 0000000
shred: 0000000: renamed to 000000
shred: 000000: renamed to 00000
shred: 00000: renamed to 0000
shred: 0000: renamed to 000
shred: 000: renamed to 00
shred: 00: renamed to 0
shred: test.txt: removed

Leave A Comment