How to Install the Linux Mode

Linux binary compatibility in FreeBSD has reached a point where it is possible to run a large fraction of Linux binaries in both a.out and ELF format. The Linux compatibility in the 2.1-STABLE branch is capable of running Linux DOOM and Mathematica; the version present in 3.3-RELEASE is vastly more capable and runs all these as well as Oracle8, WordPerfect, StarOffice, Acrobat, Quake, Abuse, IDL, and netrek for Linux and a whole host of other programs.
There are some Linux-specific operating system features that are not supported on FreeBSD. Linux binaries will not work on FreeBSD if they overly use the Linux /proc filesystem (which is different from the optional FreeBSD /proc filesystem) or i386-specific calls, such as enabling virtual 8086 mode.
Depending on which version of FreeBSD you are running, how you get Linux mode up will vary somewhat:
Installing Linux Mode in 3.0-RELEASE and later
It is no longer necessary to specify options LINUX or options COMPAT_LINUX. Linux binary compatibility is done with an KLD object (``Kernel LoaDable object'') so it can be installed on the fly without having to reboot. You will need the following things in your startup files, however:
In /etc/rc.conf, you need the following line:
  linux_enable=YES
This, in turn, triggers the following action in /etc/rc.i386:
  # Start the Linux binary compatibility if requested.
  if [ "X${linux_enable}" = X"YES" ]; then echo -n '
      linux';        linux > /dev/null 2>&1
  fi
If you want to verify that the KLD is loaded, kldstat will do that:
  % kldstat
  Id Refs Address  Size   Name
   1  2 0xc0100000 16bdb8  kernel
   7  1 0xc24db000 d000   linux.ko
If for some reason you do not want to or cannot load the Linux KLD, then statically link the binary compatibility in the kernel by adding
  options LINUX
to your kernel config file. Then run config and install the new kernel as described in the kernel configuration section.
Installing Linux Mode in 2.2.2-RELEASE and later 2.2.x versions
 It is no longer necessary to specify options LINUX or options COMPAT_LINUX. Linux binary compatibility is done with an LKM (``Loadable Kernel Module'') so it can be installed on the fly without having to reboot. You will need the following things in your startup files, however:
In /etc/rc.conf, you need the following line:
  linux_enable=YES
This, in turn, triggers the following action in /etc/rc.i386:
  # Start the Linux binary emulation if requested.
  if [ "X${linux_enable}" = X"YES" ]; then echo -n '
      linux';        linux > /dev/null 2>&1
  fi
If you want to verify that the LKM is running, modstat will do that:
  % modstat
  Type   Id Off Loadaddr Size Info   Rev Module Name
  EXEC   0  4 f09e6000 001c f09ec010  1 linux_mod
However, there have been reports that this fails on some 2.2-RELEASE and later systems. If for some reason you cannot load the Linux LKM, then statically link the Linux compatibility in the kernel by adding
  options LINUX
to your kernel config file. Then run config and install the new kernel as described in the kernel configuration section.
Installing Linux Mode in 2.1-STABLE
The GENERIC kernel in 2.1-STABLE is not configured for Linux compatibility so you must reconfigure your kernel for it. There are two ways to do this: 1. linking the binary compatibility statically in the kernel itself and 2. configuring your kernel to dynamically load the Linux loadable kernel module (LKM).
To enable Linux binary compatibility, add the following to your configuration file (c.f. /sys/i386/conf/LINT):
  options COMPAT_LINUX
If you want to run doom or other applications that need shared memory, also add the following.
  options SYSVSHM
The Linux system calls require 4.3BSD system call compatibility. So make sure you have the following.
  options "COMPAT_43"
If you prefer to statically link the binary compatibility in the kernel rather than use the loadable kernel module (LKM), then add
  options LINUX
Then run config and install the new kernel as described in the kernel configuration section.
If you decide to use the LKM you must also install the loadable module. A mismatch of versions between the kernel and loadable module can cause the kernel to crash, so the safest thing to do is to reinstall the LKM when you install the kernel.
  # cd /usr/src/lkm/linux
  # make all install
Once you have installed the kernel and the LKM, you can invoke linux as root to load the LKM.
  # linux
  Linux emulator installed
  Module loaded as ID 0
To see whether the LKM is loaded, run modstat.
  % modstat
  Type   Id Off Loadaddr Size Info   Rev
  Module Name EXEC   0  3 f0baf000 0018 f0bb4000  1 linux_emulator
You can cause the LKM to be loaded when the system boots in either of two ways. In FreeBSD 2.2.1-RELEASE and 2.1-STABLE enable it in /etc/sysconfig
  linux=YES
by changing it from NO to YES. FreeBSD 2.1 RELEASE and earlier do not have such a line and on those you will need to edit /etc/rc.local to add the following line.
  linux
Installing Linux Runtime Libraries
Installing using the linux_base port
Most Linux applications use shared libraries, so you are still not done until you install the shared libraries. It is possible to do this by hand, however, it is vastly simpler to just grab the linux_base port:
  # cd /usr/ports/emulators/linux_base
  # make all install
and you should have working Linux binary compatibility. Legend (and the mail archives :-) seems to hold that Linux mode works best with Linux binaries linked against the ZMAGIC libraries; QMAGIC libraries (such as those used in Slackware V2.0) may tend to give the Linux mode heartburn. Also, expect some programs to complain about incorrect minor versions of the system libraries. In general, however, this does not seem to be a problem.

Installing libraries manually
If you do not have the ``ports'' distribution, you can install the libraries by hand instead. You will need the Linux shared libraries that the program depends on and the runtime linker. Also, you will need to create a "shadow root" directory, /compat/linux, for Linux libraries on your FreeBSD system. Any shared libraries opened by Linux programs run under FreeBSD will look in this tree first. So, if a Linux program loads, for example, /lib/libc.so, FreeBSD will first try to open /compat/linux/lib/libc.so, and if that does not exist then it will try /lib/libc.so. Shared libraries should be installed in the shadow tree /compat/linux/lib rather than the paths that the Linux ld.so reports.
FreeBSD-2.2-RELEASE and later works slightly differently with respect to /compat/linux: all files, not just libraries, are searched for from the ``shadow root'' /compat/linux.
Generally, you will need to look for the shared libraries that Linux binaries depend on only the first few times that you install a Linux program on your FreeBSD system. After a while, you will have a sufficient set of Linux shared libraries on your system to be able to run newly imported Linux binaries without any extra work.
How to install additional shared libraries
What if you install the linux_base port and your application still complains about missing shared libraries? How do you know which shared libraries Linux binaries need, and where to get them? Basically, there are 2 possibilities (when following these instructions: you will need to be root on your FreeBSD system to do the necessary installation steps).
If you have access to a Linux system, see what shared libraries the application needs, and copy them to your FreeBSD system. Example: you have just ftp'ed the Linux binary of Doom. Put it on the Linux system you have access to, and check which shared libraries it needs by running ldd linuxxdoom:
  % ldd linuxxdoom
  libXt.so.3 (DLL Jump 3.1) => /usr/X11/lib/libXt.so.3.1.0
  libX11.so.3 (DLL Jump 3.1) => /usr/X11/lib/libX11.so.3.1.0
  libc.so.4 (DLL Jump 4.5pl26) => /lib/libc.so.4.6.29
You would need to get all the files from the last column, and put them under /compat/linux, with the names in the first column as symbolic links pointing to them. This means you eventually have these files on your FreeBSD system:
  /compat/linux/usr/X11/lib/libXt.so.3.1.0
  /compat/linux/usr/X11/lib/libXt.so.3 -> libXt.so.3.1.0
  /compat/linux/usr/X11/lib/libX11.so.3.1.0
  /compat/linux/usr/X11/lib/libX11.so.3 -> libX11.so.3.1.0
  /compat/linux/lib/libc.so.4.6.29 /compat/linux/lib/libc.so.4 -> libc.so.4.6.29
Note: Note that if you already have a Linux shared library with a matching major revision number to the first column of the ldd output, you will not need to copy the file named in the last column to your system, the one you already have should work. It is advisable to copy the shared library anyway if it is a newer version, though. You can remove the old one, as long as you make the symbolic link point to the new one. So, if you have these libraries on your system:
  /compat/linux/lib/libc.so.4.6.27
  /compat/linux/lib/libc.so.4 -> libc.so.4.6.27
and you find a new binary that claims to require a later version according to the output of ldd:
  libc.so.4 (DLL Jump 4.5pl26) -> libc.so.4.6.29
If it is only one or two versions out of date in the in the trailing digit then do not worry about copying /lib/libc.so.4.6.29 too, because the program should work fine with the slightly older version. However, if you like you can decide to replace the libc.so anyway, and that should leave you with:
  /compat/linux/lib/libc.so.4.6.29
  /compat/linux/lib/libc.so.4 -> libc.so.4.6.29
Note: The symbolic link mechanism is only needed for Linux binaries. The FreeBSD runtime linker takes care of looking for matching major revision numbers itself and you do not need to worry about it.
Configuring the ld.so --- for FreeBSD 2.2-RELEASE and later
This section applies only to FreeBSD 2.2-RELEASE and later. Those running 2.1-STABLE should skip this section.
Finally, if you run FreeBSD 2.2-RELEASE you must make sure that you have the Linux runtime linker and its config files on your system. You should copy these files from the Linux system to their appropriate place on your FreeBSD system (to the /compat/linux tree):
  /compat/linux/lib/ld.so
  /compat/linux/etc/ld.so.config
If you do not have access to a Linux system, you should get the extra files you need from various ftp sites. Information on where to look for the various files is appended below. For now, let us assume you know where to get the files.
Retrieve the following files (all from the same ftp site to avoid any version mismatches), and install them under /compat/linux (i.e. /foo/bar is installed as /compat/linux/foo/bar):
  /sbin/ldconfig
  /usr/bin/ldd
  /lib/libc.so.x.y.z
  /lib/ld.so
ldconfig and ldd do not necessarily need to be under /compat/linux; you can install them elsewhere in the system too. Just make sure they do not conflict with their FreeBSD counterparts. A good idea would be to install them in /usr/local/bin as ldconfig-linux and ldd-linux.
Create the file /compat/linux/etc/ld.so.conf, containing the directories in which the Linux runtime linker should look for shared libraries. It is a plain text file, containing a directory name on each line. /lib and /usr/lib are standard, you could add the following:
  /usr/X11/lib
  /usr/local/lib
When a Linux binary opens a library such as /lib/libc.so the Linux ABI support maps the name to /compat/linux/lib/libc.so internally. All Linux libraries should be installed under /compat/linux (e.g. /compat/linux/lib/libc.so, /compat/linux/usr/X11/lib/libX11.so, etc.) in order for the Linux ABI loader to find them.
Those running FreeBSD 2.2-RELEASE should run the Linux ldconfig program.
  # cd /compat/linux/lib
  # /compat/linux/sbin/ldconfig
ldconfig is statically linked, so it does not need any shared libraries to run. It creates the file /compat/linux/etc/ld.so.cache which contains the names of all the shared libraries and should be rerun to recreate this file whenever you install additional shared libraries.
On 2.1-STABLE do not install /compat/linux/etc/ld.so.cache or run ldconfig; in 2.1-STABLE the syscalls are implemented differently and ldconfig is not needed or used.
You should now be set up for Linux binaries which only need a shared libc. You can test this by running the Linux ldd on itself. Supposing that you have it installed as ldd-linux, it should produce something like:
  # ldd-linux `which ldd-linux`
  libc.so.4 (DLL Jump 4.5pl26) => /lib/libc.so.4.6.29
This being done, you are ready to install new Linux binaries. Whenever you install a new Linux program, you should check if it needs shared libraries, and if so, whether you have them installed in the /compat/linux tree. To do this, you run the Linux version ldd on the new program, and watch its output. ldd (see also the manual page for ldd(1)) will print a list of shared libraries that the program depends on, in the form majorname (jumpversion) => fullname.
If it prints not found instead of fullname it means that you need an extra library. The library needed is shown in majorname and will be of the form libXXXX.so.N. You will need to find a libXXXX.so.N.mm on a Linux ftp site, and install it on your system. The XXXX (name) and N (major revision number) should match; the minor number(s) mm are less important, though it is advised to take the most recent version.
Installing Linux ELF binaries
ELF binaries sometimes require an extra step of ``branding''. If you attempt to run an unbranded ELF binary, you will get an error message like the following;
  % ./my-linux-elf-binary
  ELF binary type not known
  Abort
To help the FreeBSD kernel distinguish between a FreeBSD ELF binary from a Linux binary, use the brandelf(1) utility.
  % brandelf -t Linux my-linux-elf-binary
The GNU toolchain now places the appropriate branding information into ELF binaries automatically, so you should be needing to do this step increasingly rarely in future.
Configuring the host name resolver
If DNS does not work or you get the messages
  resolv+: "bind" is an invalid keyword resolv+:
  "hosts" is an invalid keyword
then you need to configure a /compat/linux/etc/host.conf file containing:
  order hosts, bind
  multi on
where the order here specifies that /etc/hosts is searched first and DNS is searched second. When /compat/linux/etc/host.conf is not installed linux applications find FreeBSD's /etc/host.conf and complain about the incompatible FreeBSD syntax. You should remove bind if you have not configured a name-server using the /etc/resolv.conf file.
Lastly, those who run 2.1-STABLE need to set an the RESOLV_HOST_CONF environment variable so that applications will know how to search the host tables. If you run FreeBSD 2.2-RELEASE or later, you can skip this. For the /bin/csh shell use:
  % setenv RESOLV_HOST_CONF /compat/linux/etc/host.conf
For /bin/sh use:
  % RESOLV_HOST_CONF=/compat/linux/etc/host.conf; export RESOLV_HOST_CONF
Finding the necessary files
Note: The information below is valid as of the time this document was written, but certain details such as names of ftp sites, directories and distribution names may have changed by the time you read this.
Linux is distributed by several groups that make their own set of binaries that they distribute. Each distribution has its own name, like ``Slackware'' or ``Yggdrasil''. The distributions are available on a lot of ftp sites. Sometimes the files are unpacked, and you can get the individual files you need, but mostly they are stored in distribution sets, usually consisting of subdirectories with gzipped tar files in them. The primary ftp sites for the distributions are:
sunsite.unc.edu:/pub/Linux/distributions
tsx-11.mit.edu:/pub/linux/distributions
Some European mirrors:
ftp.luth.se:/pub/linux/distributions
ftp.demon.co.uk:/pub/unix/linux
src.doc.ic.ac.uk:/packages/linux/distributions
For simplicity, let us concentrate on Slackware here. This distribution consists of a number of subdirectories, containing separate packages. Normally, they are controlled by an install program, but you can retrieve files ``by hand'' too. First of all, you will need to look in the contents subdirectory of the distribution. You will find a lot of small text files here describing the contents of the separate packages. The fastest way to look something up is to retrieve all the files in the contents subdirectory, and grep through them for the file you need. Here is an example of a list of files that you might need, and in which contents-file you will find it by grepping through them:

So, in this case, you will need the packages ldso, shlibs, xf_lib and oldlibs. In each of the contents-files for these packages, look for a line saying PACKAGE LOCATION, it will tell you on which ``disk'' the package is, in our case it will tell us in which subdirectory we need to look. For our example, we would find the following locations:

The locations called ``diskXX'' refer to the slakware/XX subdirectories of the distribution, others may be found in the contrib subdirectory. In this case, we could now retrieve the packages we need by retrieving the following files (relative to the root of the Slackware distribution tree):
slakware/a2/ldso.tgz
slakware/a2/shlibs.tgz
slakware/x6/oldlibs.tgz
slakware/x9/xf_lib.tgz
Extract the files from these gzipped tarfiles in your /compat/linux directory (possibly omitting or afterwards removing files you do not need), and you are done.
See also: ftp://ftp.FreeBSD.org/pub/FreeBSD/2.0.5-RELEASE/xperimnt/linux-emu/README and /usr/src/sys/i386/ibcs2/README.iBCS2

How does the Linux mode work?
This section is based heavily on an e-mail written to the <freebsd-chat@FreeBSD.org> mailing list, written by Terry Lambert <tlambert@primenet.com> (Message ID: <199906020108.SAA07001@usr09.primenet.com>).
FreeBSD has an abstraction called an ``execution class loader''. This is a wedge into the execve(2) system call.
What happens is that FreeBSD has a list of loaders, instead of a single loader with a fallback to the #! loader for running any shell interpreters or shell scripts.
Historically, the only loader on the UNIX platform examined the magic number (generally the first 4 or 8 bytes of the file) to see if it was a binary known to the system, and if so, invoked the binary loader.
If it was not the binary type for the system, the execve(2) call returned a failure, and the shell attempted to start executing it as shell commands.
The assumption was a default of ``whatever the current shell is''.
Later, a hack was made for sh(1) to examine the first two characters, and if they were :\n, then it invoked the csh(1) shell instead (we believe SCO first made this hack).
What FreeBSD does now is go through a list of loaders, with a generic #! loader that knows about interpreters as the characters which follow to the next whitespace next to last, followed by a fallback to /bin/sh.
For the Linux ABI support, FreeBSD sees the magic number as an ELF binary (it makes no distinction between FreeBSD, Solaris, Linux, or any other OS which has an ELF image type, at this point).
The ELF loader looks for a specialized brand, which is a comment section in the ELF image, and which is not present on SVR4/Solaris ELF binaries.
For Linux binaries to function, they must be branded as type Linux; from brandelf(1):
  # brandelf -t Linux file
When this is done, the ELF loader will see the Linux brand on the file.
When the ELF loader sees the Linux brand, the loader replaces a pointer in the proc structure. All system calls are indexed through this pointer (in a traditional UNIX system, this would be the sysent[] structure array, containing the system calls). In addition, the process flagged for special handling of the trap vector for the signal trampoline code, and sever other (minor) fix-ups that are handled by the Linux kernel module.
The Linux system call vector contains, among other things, a list of sysent[] entries whose addresses reside in the kernel module.
When a system call is called by the Linux binary, the trap code dereferences the system call function pointer off the proc structure, and gets the Linux, not the FreeBSD, system call entry points.
In addition, the Linux mode dynamically reroots lookups; this is, in effect, what the union option to FS mounts ( not the unionfs!) does. First, an attempt is made to lookup the file in the /compat/linux/original-path directory, then only if that fails, the lookup is done in the /original-path directory. This makes sure that binaries that require other binaries can run (e.g., the Linux toolchain can all run under Linux ABI support). It also means that the Linux binaries can load and exec FreeBSD binaries, if there are no corresponding Linux binaries present, and that you could place a uname(1) command in the /compat/linux directory tree to ensure that the Linux binaries could not tell they were not running on Linux.
In effect, there is a Linux kernel in the FreeBSD kernel; the various underlying functions that implement all of the services provided by the kernel are identical to both the FreeBSD system call table entries, and the Linux system call table entries: file system operations, virtual memory operations, signal delivery, System V IPC, etc... The only difference is that FreeBSD binaries get the FreeBSD glue functions, and Linux binaries get the Linux glue functions (most older OS's only had their own glue functions: addresses of functions in a static global sysent[] structure array, instead of addresses of functions dereferenced off a dynamically initialized pointer in the proc structure of the process making the call).
Which one is the native FreeBSD ABI? It does not matter. Basically the only difference is that (currently; this could easily be changed in a future release, and probably will be after this) the FreeBSD glue functions are statically linked into the kernel, and the Linux glue functions can be statically linked, or they can be accessed via a kernel module.
Yeah, but is this really emulation? No. It is an ABI implementation, not an emulation. There is no emulator (or simulator, to cut off the next question) involved.
So why is it sometimes called ``Linux emulation''? To make it hard to sell FreeBSD! 8-). Really, it is because the historical implementation was done at a time when there was really no word other than that to describe what was going on; saying that FreeBSD ran Linux binaries was not true, if you did not compile the code in or load a module, and there needed to be a word to describe what was being loaded---hence ``the Linux emulator''.

© 1997 - 20013 Defcon1, www.defcon1.org , Copyrights for all materials on this web site are held by the individual authors, artists, photographers or creators. Materials may not be reproduced or otherwise distributed without permission of www.defcon1.org and the content's original author.