From bfe04cc037a15676b173fe8214909d45551ad42c Mon Sep 17 00:00:00 2001 From: Minijackson Date: Sun, 29 Sep 2019 18:41:06 +0200 Subject: Add first very WIP version --- slides.md | 717 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 717 insertions(+) create mode 100644 slides.md (limited to 'slides.md') diff --git a/slides.md b/slides.md new file mode 100644 index 0000000..cb61c9c --- /dev/null +++ b/slides.md @@ -0,0 +1,717 @@ +--- +title: NixOS for Embedded systems +author: Rémi Nicole +date: 2019-10-09 +slide-level: 2 +aspectratio: 169 +theme: metropolis +toc: true +highlightstyle: breezedark +lang: en-US + +header-includes: | + \usepackage{csquotes} + \usepackage{pgfpages} + \setbeameroption{show notes on second screen=right} + + \usecolortheme{owl} + \setbeamercolor{section in toc}{ + use=normal text, + fg=normal text.fg + } + \setbeamercolor{subsection in toc}{ + use=normal text, + fg=normal text.fg + } + + \usepackage{fvextra} +--- + +# Projects and concepts + +## {.standout} + +Heads Up + +::: notes + +- This is not a "what are you doing not using this" talk +- We will see some drawbacks that makes these tools not suitable for production + yet. +- This is more a call to try, tinker, experiment, contribute to the project so + it would be production ready + +::: + +## Definitions + +NixOS + +: Distribution + +Nix + +: Source-based package manager + +Nixpkgs + +: "Standard library" + +::: notes + +- All 3 are useful in the Embedded system context + +- "Standard library" is quoted, because it contains: + + - Packages + - Functions + - Compiling instructions for multiple languages / tools + - NixOS modules and options + +- NixOS is more of a "complete application" of the Nix package manager's + philosophy, implemented inside Nixpkgs + +::: + +## Other projects + +Hydra + +: Continuous Integration server based on Nix + +NixOps + +: NixOS cloud deployment tool + +Disnix + +: Distributed service deployment toolset + +::: notes + +Hydra + +: Can be used to produce a binary cache. Because Nix is deterministic, it is + like deploying automated Debian repository. No more need to compile Chromium + on each individual developer machine. + +NixOps + +: Like Amazon EC2 + +Disnix + +: More like Chef / Puppet for multiple hosts + +::: + +# The Nix language + +## Nix---overview + +- Source based +- Functional + - A package is a function that returns instructions on how to build a path + - Each output is in its own directory forever + `/nix/store/-/...` + - You can have multiple versions of the same package +- Binary caches +- Dynamically typed +- Lazy + +::: notes + +- Based on the ML family + +- A package is a function that depends on its dependencies, compile time + options, etc. + +- Most of the time, "how to build a path" implies building the package, and + then installing into that path + +- The hash is the sha256 of the configuration of the package, meaning the result +of the "package function". + +- A package which has the same hash does **not** need to be recompiled + +- When you have multiple versions of the same package, they are under different + store paths, and it can mean + - different upstream version + - different compilation options + - different applied patches + - different anything that would make the files in the output path different + +**TODO**: have a section to show off how to *use* different versions + +::: + +## Nix---Process + +- Use the Nix language to write derivations + + Derivation + : Data structure that describes how to create an output path (file / directory) + +- The derivation is compiled into a `.drv` file. +- Realising a derivation creates the output path +- The closure of an output path is the path and all its dependencies + +::: notes + +- Most of the time the output path is a directory which contains an FHS-like + structure + +- You can think of the `.drv` file as the result of the evaluation of your + `.nix` files + +- In simpler terms, because a package is a function, the `.drv` is the result + of that function + +::: + +## Nix---Example output paths + +``` +/nix/store/8is5yfpd095i8pcg71pb9wxv6y6d4gfv-openssh-7.9p1 +├── bin +│   ├── ssh +│   └── ... +├── etc +│   └── ssh +│   ├── ssh_config +│   └── ... +└── share + └── man + └── ... +/nix/store/chdjidjcmjs610024chncbin4bx211f2-asound.conf +/nix/store/486r3d12gc042yric302jg14in7j3jwm-i3.conf +``` + +## Nix---Language (Types) + +```nix +{ + int = 1; + boolean = true; + string = "hello ${world}!"; + "attribute set" = { a = 1; b = 2; }; + list = [ 1 2 "hello" ]; + function = a: builtins.toString a; + "function w/ named parameters" = { a, b, c ? "default"}: a; +} +``` + +::: notes + +- There's also the "derivation", which is his own type. + +- Function are just another type, they have a name when assigned to a variable + +- Functions only take one parameter. To have more than one parameter, you write +a function that returns a function ("currying"), or use named parameters. + +::: + +## Nix---Language (Control Flow) + + +:::::: {.columns} +::: {.column witdh="60%"} + +```nix +let + a = 1; + b = a + 1; + c = var: "hello ${var}!"; +in + { + d = if a == 1 then 2 else 3; + e = c "world"; + } +``` + +::: +::: {.column witdh="40%"} + +Gives: + +```nix +{ + d = 2; + e = "hello world!"; +} +``` + +::: +:::::: + +::: notes + +- This is a functional language, so every expression **must** return a value: + a `let` returns a value, an `if` returns a value so it must have an `else` + clause, etc. + +- Implies that every time you can input a value, you can input a control flow + expression + +- Functions do not need parentheses nor commas + - You can wrap everything around parentheses to force the order + - It kind of is like Lisp languages but the parentheses are optional + +::: + +## Nix---Language (Control Flow continued) + + +:::::: {.columns} +::: {.column witdh="60%"} + +```nix +let + a = 1; + b = a + 1; + c = { d = 42; inherit b; }; +in + rec { + inherit (c) b; + e = with c; d + 3; + f = e + 1; + } +``` + +::: +::: {.column witdh="40%"} + +Gives: + +```nix +{ + b = 2; + e = 45; + f = 46; +} +``` + +::: +:::::: + +::: notes + +- If we want to do it almost like the Nix interpreter, we start by the end, and + look for what we want, as needed. + +::: + +## Puzzle + +What does that do? + +```nix +rec { + a = { + inherit a; + }; +} +``` + +::: notes + +- Because the language is lazy, the question we have to ask is "What do we want + from that attribute set" + +::: + +# How 2 perfect packaging with Nixpkgs + +## An example Nixpkgs derivation + +```nix +{ stdenv, fetchurl }: + +stdenv.mkDerivation rec { + name = "hello-${version}"; + version = "2.10"; + + src = fetchurl { + url = "mirror://gnu/hello/${name}.tar.gz"; + sha256 = "0ssi1wpaf7plaswqqjwigppsg5fyh99vdlb9kzl7c9lng89ndq1i"; + }; + + doCheck = true; +``` + +::: notes + +- `stdenv` (standard environment) is a collection of packages: contains + a C compiler, autotools, make, etc. +- `stdenv` also contains the function `mkDerivation` which automates a lot of + package building +- In this example, the package is build using the standard `./configure; make; + make install` and in this case, also `make test` +- You **have** to specify the `sha256` in the `fetchurl` function to make the + network access deterministic. + - It would invalidate a lot of Nix's assumptions if the upstream mirror + changed the tarball + +::: + +--- + +```nix + meta = with stdenv.lib; { + description = "A program that produces a familiar, friendly greeting"; + longDescription = '' + GNU Hello is a program that prints "Hello, world!" when you run it. + It is fully customizable. + ''; + homepage = "https://www.gnu.org/software/hello/manual/"; + license = licenses.gpl3Plus; + maintainers = [ maintainers.eelco ]; + platforms = platforms.all; + }; +} +``` + +## Other examples {.shrink} + + +:::::: {.columns} +::: {.column} + +```nix +{ stdenv, meson, ninja, pkgconfig +, qtbase, qtconnectivity }: + +stdenv.mkDerivation rec { + pname = "setup-poc"; + version = "0.1.0"; + nativeBuildInputs = + [ meson ninja pkgconfig ]; + buildInputs = + [ qtbase qtconnectivity ]; + mesonBuildType = "debug"; + src = ./.; +} +``` + +::: +::: {.column} + +```nix +derive2 { + name = "ggplot2"; + version = "3.2.0"; + sha256 = "1cvk9pw..."; + depends = [ digest gtable lazyeval + MASS mgcv reshape2 rlang + scales tibble viridisLite + withr ]; +}; +``` + +::: +:::::: + +::: notes + +- In the meson derivation, you don't need to specify a sha256 sum for the + source: because it is local, Nix will integrate recursively the local files + into the derivation's final hash. + - This means: if a file changes, the hash changes, and it is another + package, in another path in the store + - This also means: if a file changes, Nix will know that it has to + recompile this package. And because this package is passed as an argument + to other packages, it will also recompile every packages that depends on + it. + +- **TODO**: Explain setup-hooks, and not only for build managers + + +::: + +## How do you call the function + +default.nix: + +```nix +let + pkgs = import {}; +in + pkgs.callPackage ./derivation.nix {} +``` + +We run: + +```bash +nix build --file default.nix +``` + +::: notes + +- Most of the time we use a function with default parameters, but this is for + simplicity's sake +- `callPackage` will pass the dependencies as arguments to the derivation + function +- `--file default.nix` is the default + +::: + +--- + +If we have derivation.nix: + +```nix +{ writeShellScriptBin }: + +writeShellScriptBin "myScript" "echo 'Hello, World!'" +``` + +We get as output path: + +``` +./result/ +└── bin +    └── myScript +``` + +``` +$ ./result/bin/myScript +Hello, World! +``` + +::: notes + +- `writeShellScriptBin` is a commodity function, called a "trivial builder" + + +::: + +--- + +``` +$ ls -l result +result -> /nix/store/a7db5d4v5b2pxppl8drb30ljx9z0kwg0-myScript +``` + +. . . + +`./result/bin/myScript` is: + +```bash +#!/nix/store/cinw572b38aln37glr0zb8lxwrgaffl4-bash-4.4-p23/bin/bash +echo 'Hello, World!' +``` + +::: notes + +- Everything needed by Nix or produced by Nix must be in the store +- Prevents contamination from the environment +- Build is in a chroot, with stripped env + +--- + +- This derivation depends on a specific version of Bash (because every final + dependency is specific) +- In Nix's terms, this specific Bash is in the closure of the "myScript" + derivation's output path + +::: + +## Using different versions of the same package---Generic + +```bash +#! /nix/store/...-bash-4.4-p23/bin/bash -e +export PATH='/nix/store/...-python-2.7.16/bin: + /nix/store/...-glxinfo-8.4.0/bin: + /nix/store/...-xdpyinfo-1.3.2/bin'${PATH:+':'}$PATH +export LD_LIBRARY_PATH='/nix/store/...-curl-7.64.0/lib: + /nix/store/...-systemd-239.20190219-lib/lib: + /nix/store/...-libmad-0.15.1b/lib: + /nix/store/...-libvdpau-1.1.1/lib: + ...'${LD_LIBRARY_PATH:+':'}$LD_LIBRARY_PATH +exec -a "$0" "/nix/store/...-kodi-18.1/bin/.kodi-wrapped" \ + "${extraFlagsArray[@]}" "$@" +``` + +::: notes + +- This is actually called by another wrapper who tells Kodi where to find its + data + +::: + +## Using different versions of the same package---ELF + +``` +$ readelf -d coreutils +... +Bibliothèque partagée: [librt.so.1] +Bibliothèque partagée: [libpthread.so.0] +Bibliothèque partagée: [libacl.so.1] +... +Bibliothèque runpath:[ + /nix/store/...-acl-2.2.53/lib: + /nix/store/...-attr-2.4.48/lib: + /nix/store/...-openssl-1.0.2t/lib: + /nix/store/...-glibc-2.27/lib +] +... +``` + +## Using different versions of the same package---Python + +```python +# imports... +sys.argv[0] = '/nix/store/...-carla-2.0.0/share/carla/carla' +functools.reduce( + lambda k, p: site.addsitedir(p, k), + [ + '/nix/store/...-python3.7-rdflib-4.2.2/lib/python3.7/site-packages', + '/nix/store/...-python3.7-isodate-0.6.0/lib/python3.7/site-packages', + '/nix/store/...-python3.7-six-1.12.0/lib/python3.7/site-packages', + # ... + ], + site._init_pathinfo()) +# ... +``` + +::: notes + +- Because these paths are in the closure of the app, they are guaranteed by Nix + to be there + +::: + +# NixOS + +## How do you make a Linux distribution out of that? + +- A distribution is a bunch of files +- In the end, the Nix package manager creates files +- Let's use Nix to create *every* (non user data) file + +## Adding yourself to the environment---Symbolic links + +``` +$ ls -l /etc/static/ssh/ssh_config +/etc/static/ssh/ssh_config -> /nix/store/...-etc-ssh_config +``` + +--- + +``` +$ systemctl cat sshd.service + +# /nix/store/...-unit-sshd.service/sshd.service +# ... +[Service] +Environment="LD_LIBRARY_PATH=..." +Environment="PATH=..." + +ExecStart=/nix/store/...-openssh-7.9p1/bin/sshd -f /etc/ssh/sshd_config +ExecStartPre=/nix/store/...-unit-script-sshd-pre-start +KillMode=process +Restart=always +Type=simple +``` + +::: notes + +- The service file is in the nix store, especially in its own path! +- All linked into `/etc/systemd` so systemd can find them +- The maintainers could have put a nix store path for the sshd_config + - The decision was made for sysadmins to be able to quick look their config + (`nixpkgs/pull/41744`) +- The pre start script generates host key if non-existent + + +::: + +## Adding yourself to the environment---Environment variables + +``` +$ echo $XDG_DATA_DIRS +/run/opengl-driver/share: +/run/opengl-driver-32/share: +/home/minijackson/.nix-profile/share: +/etc/profiles/per-user/minijackson/share: +/nix/var/nix/profiles/default/share: +/run/current-system/sw/share +``` + +::: notes + +- Inside these dirs are symbolic links +- Environment variables with hardcoded nix store paths are quite rare in the + user environment + - When possible, we usually do this in the packaging + + +::: + +## Adding yourself to the environment---Tool specific + +TODO: find a tool + +## How we do it + +::: notes + +- We talked about how it is possible for NixOS to do it, now we talk about how + us devs write the code + + +::: + +## Assertions + +``` +Failed assertions: +- services.xserver.xkbOptions = "eurosign:e" is useless on fr/ch + layouts and plays badly with bépo +``` + +# The embedded world + +## TODO + +- [x] Use good Markdown / Beamer template +- [ ] Pinning repo version +- [x] How to use different versions +- [ ] How to build an image +- [ ] Add some images to temporise the talk +- [ ] Talk about service tests!!! + - [ ] Single-node + - [ ] Multi-node +- [ ] Add derivation example with compilation options +- [ ] Add intro? about problems that I had in Buildroot (non-determinism, no + auto-recompile) +- [ ] Talk about nix-shell +- [ ] Talk about nixops +- [ ] Talk about choosing generation at boot +- [ ] Have a functional programming intro +- [ ] You **don't** need to run NixOS to develop with Nix +- [ ] https://r13y.com/ +- [ ] Drawbacks + - [ ] Harder to compile anything, especially packages with bad build systems + - [ ] If one package doesn't compile, harder to compile the system, since + it's a dependency of the system closure + - [ ] You need deeper understanding of the ecosystem to package random + programs + - [ ] Sometimes need to patch software to bypass hardcoded paths (e.g. + locale archive) + - [ ] For now, only world readable store, makes it kinda harder for + passwords and other secrets + - Documentation / developer experience is sub-optimal +- [ ] A burnable image is just a file that depends on every package of your + system + - [ ] instead of depending manually on each package, we depend on the + top-level system closure (output and all dependencies) + +## The End {.standout} + +That's all folks! + +. . . + +Questions? -- cgit v1.2.3