$Id: README,v 1.1 2006/04/12 21:43:03 mmenal Exp $ Memory based filesystem translators for the Hurd Copyright (C) 2000 Farid Hajji These programs are free software; you can redistribute them and/or modify them under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2, or (at your option) any later version. These programs are distributed in the hope that they will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with these programs; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA Memory based filesystems: ------------------------- Unlike regular filesystems, the contents of memory-based filesystems are stored entirely in memory (think ramdisk). This is mainly for performance reasons, but also to provide a testbed for new filesystem implementations in case no free partition is available. The programs in the directory provide implementations for memory-based filesystems in the Hurd. memfs-1: (filesystems for the lazy and impatient) -------- A file in the Hurd is nothing more than a port to a translator that provides the necessary semantics. For this reason, one can basically create a big file (e.g. with dd), write a filesystem in it and settrans the corresponding filesystem translator on it: dd if=/dev/zero of=/path/to/bigfile bs= count=1 mkfs.ext2 /path/to/bigfile settrans -ac /path/to/target /hurd/ext2fs /path/to/bigfile Accessing a file in /path/to/target is a multistage process: client -> /hurd/ext2fs[bigfile] -> /hurd/ufs[/path/to] -> storeio[/dev/hd0s1] <- <- <- memfs-1 is a translator that provides a memory-based file of fixed size. This file can, just like bigfile, contain a regular filesystem: settrans -ac /path/to/memfile /path/to/memfs-1 -s mkfs.ext2 /path/to/memfile settrans -ac /path/to/target /hurd/ext2fs /path/to/memfile Again, accessing files in /path/to/target involves many translators, but fewer than in the previous case: client -> /hurd/ext2fs[memfile] -> /path/to/memfs-1 <- <- memfs-1 allocates an internal buffer on startup (currently via mmap()) and handles every trapped read(), write() and seek() operation by copying data in/out of this buffer. Physical accesses to a disk are completely avoided, as long as memfs-1 and the external filesystem translator are resident in memory (else: accesses to swap disk[s]). memfs-1 is a simple modification to the multithreaded hello-mt translator written by the Free Software Foundation, Inc. As such, it is not robust either, since many trivfs callbacks are not (yet) implemented. For example, it is not possible to change the size of the memfile. Doing so (e.g. try this with vi) will result in an EIEIO error. Using memfs-1: -------------- ### unpack and install $ tar -zxvf memfs-1.0.tar.gz $ cd mem-fs $ make cc -Wall -g -c memfs-1.c -o memfs-1.o cc -Wall -g -o memfs-1 memfs-1.o -lthreads -lports -ltrivfs -lfshelp cc -Wall -g -c test-memfs-1.c -o test-memfs-1.o cc -Wall -g -o test-memfs-1 test-memfs-1.o ### setup memfs-1 as a translator for xxx $ make start-memfs-1 settrans -ac xxx ./memfs-1 --size 40 --debug 10 ### memfs-1 is really running $ ps ax | grep memfs farid 192 - So 0:00.05 ./memfs-1 --size 40 --debug 10 ### debug output is in .out: $ cat .out ./memfs-1 [2000-12-19 19:04:57]: parse_opt ./memfs-1 [2000-12-19 19:04:57]: parse_opt ./memfs-1 [2000-12-19 19:04:57]: parse_opt ./memfs-1 [2000-12-19 19:04:57]: parse_opt ### The memfs-1 translated memory file xxx is there: $ ls -l xxx -rw-rw-rw- 1 farid farid 40 Dec 19 19:04 xxx ### The previous ls(1) triggered the following events: $ cat .out ./memfs-1 [2000-12-19 19:06:53]: open_hook ./memfs-1 [2000-12-19 19:06:53]: close_hook ### Let's write 10 bytes in xxx at offset 5: $ ./test-memfs-1 xxx 5 I am now at offset 5 written 10 bytes I am now at offset 0 got 40 bytes: [.....0123456789.........................] ### This triggered the following events: $ cat .out ./memfs-1 [2000-12-19 19:08:25]: open_hook ./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_seek(offs=5) ./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_write(offs=-1, data_len=10) ./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_seek(offs=0) ./memfs-1 [2000-12-19 19:08:25]: trivfs_S_io_read(offs=-1, amount=40) ./memfs-1 [2000-12-19 19:08:25]: close_hook ### Let's write near the end of the file, effectively overflowing it: $ ./test-memfs-1 xxx 37 I am now at offset 37 written 3 bytes I am now at offset 0 got 40 bytes: [.....0123456789......................012] ### Looking at the new events again: $ cat .out ./memfs-1 [2000-12-19 19:09:33]: open_hook ./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_seek(offs=37) ./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_write(offs=-1, data_len=10) ./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_seek(offs=0) ./memfs-1 [2000-12-19 19:09:33]: trivfs_S_io_read(offs=-1, amount=40) ./memfs-1 [2000-12-19 19:09:33]: close_hook ### Writing past the end of the file is silently rejected: $ ./test-memfs-1 xxx 40 I am now at offset 40 written 0 bytes I am now at offset 0 got 40 bytes: [.....0123456789......................012] $ ./test-memfs-1 xxx 500 I am now at offset 40 written 0 bytes I am now at offset 0 got 40 bytes: [.....0123456789......................012] $ ./test-memfs-1 xxx -2 I am now at offset 40 written 0 bytes I am now at offset 0 got 40 bytes: [.....0123456789......................012] ### We don't need the memfs-1 translator anymore: $ make stop-memfs-1 settrans -fg xxx ### trivfs_goaway() was really called: $ cat .out ./memfs-1 [2000-12-19 19:10:29]: open_hook ./memfs-1 [2000-12-19 19:10:29]: close_hook ./memfs-1 [2000-12-19 19:10:29]: trivfs_goaway ### translator memfs-1 is away (old time): $ ls -l xxx -rw-r--r-- 1 farid farid 0 Dec 19 19:04 xxx Using memfs-1 as a filesystem translator: ----------------------------------------- ### Let's create a big memory file: $ settrans -ac xxx ./memfs-1 --size 67108864 --debug 2 ### We can distinguish this translator in 'ps' ### but due to the limit in MAXSIZE (see memfs-1.c) ### the file was truncated to 30MB: $ ps ax | grep memfs farid 385 - So 0:00.90 ./memfs-1 --size 67108864 --debug 2 $ ls -l xxx -rw-rw-rw- 1 farid farid 31457280 Dec 19 19:04 xxx ### Now, create an ext2fs filesystem in xxx: $ /sbin/mke2fs xxx mke2fs 1.19, 13-Jul-2000 for EXT2 FS 0.5b, 95/08/09 Filesystem label= OS type: GNU/Hurd Block size=1024 (log=0) Fragment size=1024 (log=0) 7680 inodes, 30720 blocks 1536 blocks (5.00%) reserved for the super user First data block=1 4 block groups 8192 blocks per group, 8192 fragments per group 1920 inodes per group Superblock backups stored on blocks: 8193, 24577 Writing inode tables: done Writing superblocks and filesystem accounting information: done ### The new filesystem can now be mounted as usual: $ settrans -ac newdir /hurd/ext2fs xxx $ ls -ld newdir drwxr-xr-x 3 farid farid 1024 Dec 19 19:46 newdir $ /bin/ls -l newdir total 12 drwxr-xr-x 2 root root 12288 Dec 19 19:46 lost+found ### Filesystem newdir/ is now usable just like any other filesystem. ### Remove the translators if not needed anymore: $ settrans -fg newdir $ settrans -fg xxx One final note: memfs-1 currently _requires_ a --size parameter. Use a --debug value >=5 to get a trace in .out Suggested future work: ---------------------- memfs-1: add missing trivfs callbacks. memfs-2: like memfs-1, but uses libpager for the internal buffer. memfs-3: like memfs-1, but uses libstore (memory or task) for the buffer. memfs-4: netfs-based true filesystem with mmap() allocated internal buffers. memfs-5: netfs-based true filesystem with libpager backed internal buffers. memfs-6: netfs-based true filesystem with libstore (memory/task) for buffers. memfs-7: diskfs-based true filesystem, mmap() allocated buffers. memfs-8: diskfs-based true filesystem, libpager backed buffers. memfs-9: diskfs-based true filesystem, libstore (memory/task) for buffers. memfs-{a,b,c,...}: filesystems should export the buffer[s] via mmap() to clients to avoid copyin/copyout. (This can be done by adding a RPC so that clients can send their task control port in a request.) shm*() library : use memfs-{a,b,c...} to implement SysV-ish shared memory. Add more filesystem types (minix, vfat, ...) usable with memfs-{1,2,3}. -- Farid Hajji -- Unix Systems and Network Admin | Phone: +49-2131-67-555 Broicherdorfstr. 83, D-41564 Kaarst, Germany | farid.hajji@ob.kamp.net - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - Murphy's Law fails only when you try to demonstrate it, and thus succeeds.