% The File System --- subtitle: WTF is Linux --- [back](index.md) ## Purpose If we look at storage devices as a way to store a bunch of ones and zeroes, we can see that this is not the most humane way of storing data. At the hardware level, there is no concept of file or directories, so we need a way of organizing these ones and zeroes in such a way that we can tell "such and such series of bytes correspond to the `/home/user/file.txt` content". This is exactly what a file system is. At the core, a file system is a way of organizing data (we call this a "format", since it's a way of "formatting" data). Once we have some storage formatted for a known file system, we rely on the kernel to be able to read this file system. Then, once we know the kernel is able to read the file system, we want to tell the kernel where we would like to see it in our directory tree. This is called "[mounting](#mounts)" a file system. ## Storage devices Before we talk about the file system properly, let's talk about different kinds of storage devices. We said in the introduction that "storage devices are a way to store a bunch of ones and zeroes", but different kinds of devices have different specificities. For example, with a Hard Disk Drive (HDD), you will want to read close data in quick succession, else you would have to wait for the next rotation of the disk. In Flash Memory, used in USB Flash drives and often used in embedded systems, you cannot set a single 0 to a 1, but instead have to erase a whole "sector" to set every bit in that sector back to a 1. While these specificities might have an impact in high-end computing, or bare metal embedded systems (i.e. without a kernel), in our case, we thankfully don't have to care about any of that. Every particularity of these devices is abstracted away by "device drivers" implemented in the kernel, leaving us with a much simpler interface, like `read` and `write` system calls. ## Some well-known file systems Just as there are multiple ways of storing an image in a file (jpeg, png, bmp, etc.), there are multiple file system formats. ### FAT32 FAT32 is probably the most "compatible" file system around. USB Flash Drives usually comes pre-formatted as FAT32, and when plug a USB Flash Drive into something (computer, router, gaming console, etc.), you can be pretty sure it'll work. It is not without drawbacks though: it is prone to fragmentation, can't handle files bigger than 4 GB, and doesn't have many features that more modern file systems have (e.g. error recovery). ### ext4 ext4 is the de facto standard file system on GNU/Linux: when you install a GNU/Linux distribution, you will most likely end up with partitions formatted using ext4. ### NTFS NTFS is the standard file system used by Windows computers. To my knowledge, it is not much used elsewhere, except to have Windows compatible external drives, and support files bigger than 4 GB (so FAT32 is not a valid choice). ## Enabling file systems in the kernel As an end user of a normal GNU/Linux distributions, there is usually no need to enable the support for any file system, as the most common ones are enabled by default. As an embedded system developer however, this is useful to know how to enable, disable file systems: you already know which file system(s) you're going to use, so you can only enable those, and disable the rest. Disabling unused file systems have two main benefits: - Less code in the kernel means a smaller kernel, and "size matters" is actually true in the embedded world - Less code in the kernel also means less attack surface, so less potential vulnerabilities in the kernel To enable or disable a file system in the kernel, simply go to your [configuration](kernel.md#configuring-the-kernel), look up your file system in the "File Systems" menu, and enable or disable it. ## Anatomy of a file system Because a file system stores more that just the content of files, it needs space for this "metadata". Different file systems do this differently, and have different needs, so let's quickly look at a few examples: TODO: install the pandoc-citeproc filter ![FAT32 format](./res/fat32.jpg) The FAT32 way of doing things is the simplest: it simply has a table of files and directories at the beginning, and space afterwards for the content of files. ## Partitioning schemes We've seen how a file system works, but we're missing a crucial component of the storage story: partitions. While having an ext4 file system take up a full hard drive is possible, this is usually not done in practice. In a lot of cases, we want *several* file systems in a single hard drive. An embedded example is having two system partitions, and if booting to one partitions fails, fall back on booting on the other (in case a system upgrade broke something, for example). Like everything else in computer science, you have several ways of doing it: in a similar way to file systems, partitioning schemes are a way of formatting ones and zeroes, but instead of storing files, it stores file systems. The two most common partitioning schemes are GPT (the most modern one) and MBR (which you'll often find on older systems). TODO: reference source of image ![GPT format](./res/GUID-Partition-Table-Scheme.svg) Like file format, you'll find sections dedicated to the metadata of the partitions (i.e. name of the partitions, id of the partition, etc.), and a whole space (partition data) to put your file system. ## Block devices Now that we know how partitions and file systems work, let's see how that integrates with the GNU/Linux ecosystem. With the "everything is a file" philosophy of Linux, we can safely expect to see our disks and partitions represented as files. These special kind of files are called "block devices", and can be found in the `/dev` directory. Depending on how your disk is connected to your computer, the block device is going to be named differently in `/dev`. On most systems, hard disk drives and SSDs are connected using an [SCSI](https://en.wikipedia.org/wiki/SCSI) connection, so you will find your drives under `/dev/sda`, `/dev/sda1`, `/dev/sdb`, etc. The naming scheme goes like this: ``` ,----- Represents the whole first SCSI disk the kernel found | v /dev/sda /dev/sda1 <-- Represents the first partition of the first disk /dev/sda2 <-- Represents the second partition of the first disk ,----- Represents the whole second SCSI disk the kernel found | v /dev/sdb /dev/sdb1 <-- Represents the first partition of the second disk /dev/sdb2 <-- Represents the second partition of the second disk ``` So, if we wanted to read directly from the file `/dev/sda`, you would get exactly the bytes that are stored in your first storage device (and you could parse yourself the GPT or MBR format). But this is not what we usually want, we want to access the partitions inside the disk. For this, the kernel provides us with the files `/dev/sda1`, `/dev/sda2`, etc. Similarly, if you were to read directly from the file `/dev/sda2`, you would get the bytes that are stored in the second partition of the first disk (and you could parse yourself the format ext4, fat32, etc.) Again, we do not usually want to access the bytes inside the partition, but we want to access the files stored inside the file system of the partition! To that end, what we do is [mount](#mounts) the file system. Accessing the raw bytes of the disk or partition is often useful, however. It can be used to list all the partitions in a disk, resize partitions, or get some information of a given file system. To read or modify partitions, you can use the `parted` command-line tool, or `gparted` graphical program: ```bash user@host:~$ sudo parted -l Model: Thumb Drive (scsi) Disk /dev/sdb: 4041MB Sector size (logical/physical): 512B/512B Partition Table: gpt Number Start End Size File system Name Flags 1 17.4kB 1000MB 1000MB first 2 1000MB 4040MB 3040MB second ``` ![GParted screenshot](./res/GParted_1.0_screenshot.png) To read or modify file systems, well it depends on the file system format you want to access. To access file systems in the ext4 format, you can use programs from the [e2fsprogs project](https://en.wikipedia.org/wiki/E2fsprogs). To access file systems in the fat32 format, you can use programs from the [dosfstools project](https://directory.fsf.org/wiki/Dosfstools). TODO: create a file system ## Mounts "Mounting" a file system is the action of representing the files of a partition or disk into the directory hierarchy. For example, if the `/dev/sda1` file system is mounted, files of the first partition of the first disk will appear, and reading, writing, or changing the permissions of these files will instruct the kernel to read or modify the bytes stored in that first partition of the first disk (in the file data if reading/writing to the file, and in the file meta-data if modifying permissions, in our example). To mount a file system, we can use the `mount` command (being root is needed): ```bash # ,----------- What file system to mount # | (disk or partition) # | # | ,---- Where to mount it (a directory) # | | # vvvvvvvvv vvvv root@host:~$ mount /dev/sda1 /mnt ``` With this command, the kernel will try to auto-detect the type of file system (ext4, fat, etc.), and provide you with the files inside that file system under the given directory (in this example: `/mnt`). The directory where a file system is mounted is called a "mount point". You can also specify mount options, with the `-o` or `--options` command-line argument. Common options include: - `ro`: read-only - `rw`: read-write - `defaults`: several common options - `sync`: when a file is read or written, do it directly on the hardware - `async`: when a file is read or written, do it asynchronously For more information, see the man page `man 8 mount`. ## Special file systems One consequence of the "everything is a file" philosophy of Linux, is that not every file has to be tied to a storage device. For example, or files `/dev/sda`, ... that represent or hard drives and partitions don't have to be stored in a physical location. It's way better to create these files on boot according to the present hardware, and forget them when shutting down the machine. An often used file system like that is the `tmpfs` special file system. Like its name suggests, it is a temporary file system. Every file and directory in it is stored on memory only, and disappears on shutdown. Another commonly used temporary file system is the `devtmpfs` special file system. It is much like `tmpfs`, but instead of being created empty, all the block devices (among other things) like `sda`, `sda1`, ... are automatically created inside it. In that sense, the `/dev` directory really is just a standard mount point for the `devtmpfs` file system. Finally, two really useful file systems are the `sysfs` and `proc` special file systems. The `sysfs` is there to export information directly from the kernel. It can be related to hardware devices, drivers, etc. It is normally mounted under the `/sys` directory. The `proc` is mainly there to provide information about currently running processes. For example, doing `cat /proc//cmdline` will get you the command-line arguments of the running process with the given ``. It is normally mounted under the `/proc` directory. There are also a lot of configuration options under the `/proc/sys` directory. ## Further reading - "Design of the FAT file system" on Wikipedia : - ext4 on Wikipedia : - GPT on Wikipedia : - MBR on Wikipedia : - Using the `parted` command line tool on LinuxJourney : - Manual of the `mount` command : - Manual of `sysfs` file system : - Manual of `proc` file system :