path: root/file-system.md

% The File System

---
subtitle: WTF is Linux
---

[back](index.md)


## Purpose

If we look at storage devices as a way to store a bunch of ones and zeroes, we
can see that this is not the most humane way of storing data. At the hardware
level, there is no concept of file or directories, so we need a way of
organizing these ones and zeroes in such a way that we can tell "such and such
series of bytes corresponds to the `/home/user/file.txt` content".

This is exactly what a file system is. At the core, a file system is a way of
organizing data (we can this a "format", since it's a way of "formatting"
data).

Once we have some storage formatted for a known file system, we rely on the
kernel to be able to read this file system.

Then, once we know the kernel is able to read the file system, we want to tell
the kernel where we would like to see it in our directory tree. This is called "[mounting](#mounts)" a file system.


## Storage devices

Before we talk about the file system properly, let's talk about different kinds
of storage devices.

We said in the introduction that "storage devices are a way to store a bunch of
ones and zeroes", but different kinds of devices have different specificities.

For example, with a Hard Disk Drive (HDD), you will want to read close data in
quick succession, else you would have to wait for the next rotation of the
disk.

In Flash Memory, used in USB Flash drives and often used in embedded systems,
you cannot set a single 0 to a 1, but instead have to erase a whole "sector" to
set every bit in that sector back to a 1.

While these specificities might have an impact in high end computing, or bare
metal embedded systems (i.e. without a kernel), in our case, we thankfully
don't have to care about any of that.

Every particularity of these devices is abstracted away by "device drivers"
implemented in the kernel, leaving us with a much simpler interface, like
`read` and `write` system calls.


## Some known file systems

Just as there are multiple ways of storing an image in a file (jpeg, png, bmp,
etc.), there are multiple file system formats.

### FAT32

FAT32 is probably the most "compatible" file system around.

USB Flash Drives usually comes pre-formatted as FAT32, and when plug a USB
Flash Drive into something (computer, router, gaming console, etc.), you can be
pretty sure it'll work.

It is not without drawbacks though: it is prone to fragmentation, can't handle
files bigger than 4GB, and doesn't have many features that more modern file
systems have (e.g. error recovery).

### ext4

ext4 is the de facto standard file system on GNU/Linux: when you install a
GNU/Linux distribution, you will most likely end up with partitions formatted
using ext4.

### NTFS

NTFS is the standard file system used by Windows computers. To my knowledge, it
is not much used elsewhere, except to have Windows compatible external drives,
and support files bigger than 4GB (so FAT32 is not a valid choice).


## Enabling file systems in the kernel

As an end user of a normal GNU/Linux distributions, there is usually no need to
enable the support for any file system, as the most common ones are enabled by
default.

As an embedded system developer however, this is useful to know how to enable,
disable file systems: you already know which file system(s) you're going to
use, so you can only enable those, and disable the rest.

Disabling unused file systems have two main benefits:

- Less code in the kernel means a smaller kernel, and "size matters" is
  actually true in the embedded world
- Less code in the kernel also means less attack surface, so less potential
  vulnerabilities in the kernel

To enable or disable a file system in the kernel, simply go to your
[configuration](kernel.md#configuring-the-kernel), look up your file system in
the "File Systems" menu, and enable or disable it.

## Usual format of a file system

Because a file system stores more that just the content of files, it needs
space for this "meta data". Different file systems do this differently, and
have different needs, so let's quickly look at a few examples:

TODO: reference source of image

![FAT32 format](./res/fat32.jpg)

The FAT32 way of doing things is the simplest: it simply has a table of files
and directories at the beginning, and space afterwards for the content of
files.

## Partitioning schemes

We've seen how a file system works, but we're missing a crucial component of
the storage story: partitions.

While having an ext4 file system take up a full hard drive is possible, this is
usually not done in practice.

In a lot of cases, we want *several* file systems in a single hard drive. An
embedded example is having two system partitions, and if booting to one
partitions fails, fall back on booting on the other (in case a system upgrade
broke something, for example).

Like everything else in computer science, you have several ways of doing it: in
a similar to file systems, partitioning schemes are a way of formatting ones
and zeroes, but instead of storing files, it stores file systems.

The two most common partitioning schemes are GPT (the most modern one) and MBR
(which you'll often find on older systems).

TODO: reference source of image

![GPT format](./res/GUID-Partition-Table-Scheme.svg)


Like file format, you'll find sections dedicated to the meta data of the
partitions (i.e. name of the partitions, id of the partition, etc.), and
a whole space (partition data) to put your file system.

## Block devices


Now that we know how partitions and file systems work, let's see how that
integrates with the GNU/Linux ecosystem.

With the "everything is a file" philosophy of Linux, we can safely expect to
see our disks and partitions represented as files. These special kind of files
are called "block devices", and can be found in the `/dev` directory.

Depending on how your disk is connected to your computer, the block device is
going to be named differently in `/dev`.

On most systems, hard disk drives and SSDs are connected using an
[SCSI](https://en.wikipedia.org/wiki/SCSI) connection, so you will find your
drives under `/dev/sda`, `/dev/sda1`, `/dev/sdb`, etc.

The naming scheme goes like this:

```
       ,----- Represents the whole first SCSI disk the kernel found
       |
       v
/dev/sda
/dev/sda1 <-- Represents the first  partition of the first disk
/dev/sda2 <-- Represents the second partition of the first disk

       ,----- Represents the whole second SCSI disk the kernel found
       |
       v
/dev/sdb
/dev/sdb1 <-- Represents the first  partition of the second disk
/dev/sdb2 <-- Represents the second partition of the second disk
```

So, if we wanted to read directly from the file `/dev/sda`, you would get
exactly the bytes that are stored in your first storage device (and you could
parse yourself the GPT or MBR format).

But this is not what we usually want, we want to access the partitions inside
the disk. For this, the kernel provides us with the files `/dev/sda1`,
`/dev/sda2`, etc.

In a similar fashion, if you were to read directly from the file `/dev/sda2`,
you would get the bytes that are stored in the second partition of the first
disk (and you could parse yourself the format ext4, fat32, etc.)

Again, we do not usually want to access the bytes inside of the partition, but
we want to access the files stored inside of the file system of the partition!
To that end, what we do is [mount](#mounts) the file system.

Accessing the raw bytes of the disk or partition is often useful, however. It
can be used to list all the partitions in a disk, resize partitions, or get
some information of a given file system.

To read or modify partitions, you can use the `parted` command-line tool, or
`gparted` graphical program:

```bash
user@host:~$ sudo parted -l

Model: Thumb Drive (scsi)
Disk /dev/sdb: 4041MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start   End     Size     File system  Name        Flags
 1      17.4kB  1000MB  1000MB                first
 2      1000MB  4040MB  3040MB                second
```

![GParted screenshot](./res/GParted_1.0_screenshot.png)

To read or modify file systems, well it depends on the file system format you
want to access.

To access file systems in the ext4 format, you can use programs from the
[e2fsprogs project](https://en.wikipedia.org/wiki/E2fsprogs).

To access file systems in the fat32 format, you can use programs from the
[dosfstools project](https://directory.fsf.org/wiki/Dosfstools).

TODO: create a file system

## Mounts

"Mounting" a file system is the action of representing the files of a partition
or disk into the directory hierarchy.

For example, if the `/dev/sda1` file system is mounted, files of the first
partition of the first disk will appear, and reading, writing, or changing the
permissions of these files will instruct the kernel to read or modify the bytes
stored in that first partition of the first disk (in the file data if
reading/writing to the file, and in the file meta-data if modifying
permissions, in our example).

To mount a file system, we can use the `mount` command (being root is needed):

```bash
#                      ,----------- What file system to mount
#                      |            (disk or partition)
#                      |
#                      |      ,---- Where to mount it (a directory)
#                      |      |
#                  vvvvvvvvv vvvv
root@host:~$ mount /dev/sda1 /mnt
```

With this command, the kernel will try to auto detect the type of file system
(ext4, fat, etc.), and provide you with the files inside that file system under
the given directory (in this example: `/mnt`).

The directory where a file system is mounted is called a "mount point".

You can also specify mount options, with the `-o` or `--options` command-line
argument. Common options include:

- `ro`: read-only
- `rw`: read-write
- `defaults`: several common options
- `sync`: when a file is read or written, do it directly on the hardware
- `async`: when a file is read or written, do it asynchronously

For more information, see the man page `man 8 mount`.


## Special file systems

One consequence of the "everything is a file" philosophy of Linux, is that not
every file has to be tied to a storage device.

For example, or files `/dev/sda`, ... that represent or hard drives and
partitions don't have to be stored in a physical location. It's way better to
create these files on boot according to the present hardware, and forget them
when shutting down the machine.

An often used file system like that is the `tmpfs` special file system. Like
its name suggests, it is a temporary file system. Every file and directory in
it is stored on memory only, and disappears on shutdown.

Another commonly used temporary file system is the `devtmpfs` special file
system. It is much like `tmpfs`, but instead of being created empty, all the
block devices (among other things) like `sda`, `sda1`, ... are automatically
created inside it.

In that sense, the `/dev` directory really is just a standard mount point for
the `devtmpfs` file system.

Finally, two really useful file systems are the `sysfs` and `proc` special file
systems.

The `sysfs` is there to export information directly from the kernel. It can be
related to hardware devices, drivers, etc. It is normally mounted under the
`/sys` directory.

The `proc` is mainly there to provide information about currently running
processes. For example, doing `cat /proc/<PID>/cmdline` will get you the
command-line arguments of the running process with the given `<PID>`. It is
normally mounted under the `/proc` directory.

There are also a lot of configuration options under the `/proc/sys` directory.

## Further reading

- "Design of the FAT file system" on Wikipedia
: <https://en.wikipedia.org/wiki/Design_of_the_FAT_file_system>
- ext4 on Wikipedia
: <https://en.wikipedia.org/wiki/Ext4>
- GPT on Wikipedia
: <https://en.wikipedia.org/wiki/GUID_Partition_Table>
- MBR on Wikipedia
: <https://en.wikipedia.org/wiki/Master_boot_record>
- Using the `parted` command line tool on LinuxJourney
: <https://linuxjourney.com/lesson/disk-partitioning>
- Manual of the `mount` command
: <https://linux.die.net/man/8/mount>
- Manual of `sysfs` file system
: <https://man7.org/linux/man-pages/man5/sysfs.5.html>
- Manual of `proc` file system
: <https://man7.org/linux/man-pages/man5/proc.5.html>