memtester asks for memory, linux oom crashes computer

Linux will allow one to overcommit on memory allocations. With large allocations like this, the system will run low on free memory and start to swap out old dirty (used) pages to the swap partition to free up pages for the allocation. In doing so your system will become less responsive as all the swap activity will use all the available disk bandwidth in servicing the swap activity.

So:

1. "How is it possible that memtester did not got rejected by the OS when asking for more memory than available?"

- that's because the kernel allows one to allocate more memory than is available because of the overcommit behaviour, plus it will service this with swapping. This is normal behaviour.

2. "How is it possible that memtester did not got rejected by the user process's limits?"

Reading the manual to memtester I see that it recommends running it as root; this will therefore ignore the ulimit set to your normal user id. e.g.

$ ulimit -m
2097152
$ sudo memtester 8GB 10

[ the sudo will ignore your user ulimit ]

3. I see that memtester will attempt to mlock pages into memory. Running it as root will effectively allow it to lock pages into memory so they can't be swapped out. Hence when the system runs low on memory the kernel will attempt to swap out pages of other running processes first (since memtester's pages are locked in and can't be swapped out). Eventually the kernel will run out of pages to evict and the OOM killer will kill memtester.

4. "How to prevent memtester (and any other software) to crash my system by simply greedily asking for memory?"

To stop memtester being OOM'd, run it with the just the size of available memory rather than the physical memory size. The 'free' command will show you how much memory is available, e.g. on my machine:

$ free -m
              total used free shared buff/cache available
Mem: 15769 2108 9362 274 4299 13059
Swap: 7812 0 7812

..showing there are 9362 MB of free memory, so run memtester with the 'free' memory size.

I would uninstall the program. During the boot -> Advanced options- Memory Test