$Id: webserver.txt,v 1.2 2001/09/12 09:42:05 brahe Exp $

On Wed, Aug 08, 2001 at 06:32:01PM +0200, Saad Kadhi wrote:
> I got a look at the mentioned webpage. It is scary :(. From previous
> benchmarks that I've seen, OpenBSD was a good performer but now this
> (OT: how come FreeBSD is so bad in comparison to Linux 2.4.7? I thought
> FreeBSD was the best-of-breed while tuned ?).  
> Henning, I really think some of us are holding their breath and waiting
> for your comments. How come it's that bad w/ 3Com 905B/SCSI hw ?

For first: I'm not too surprised that a linux default install performs
better than any of the others, especially OpenBSD. I'm not too familar with
the other BSDs, and I don't have experience with linux 2.4. I consider
2.2 slighly bloated, and 2.4 is a hell. 
Linux kernels are usually modularized. BSD kernels not. so it is quite clear
that a linux kernel, having loaded only the needed modules, is faster than a
GENERIC OpenBSD kernel having compiled everything supported in. I guess it's
the same for the other BSDs. For a real comparision you must build (very)
custom kernels. The default kernel is nearly 4 MB huge, the ones on my
production machines, including IPv6 support and Linux + BSD/OS binary 
emulation, are about 1.6 MB. Without IPv6 and binary emulations about 1 MB 
should be reachable.
I'm not really sure why OpenBSD is _that_ worse than Net/FreeBSD and linux
2.2, but it's clear why these and OpenBSD are worse than linux 2.4 here. The
test opens a massive number of parallel httpds. (btw, your httpd.conf is
_not_ the default one. The default one does not state "MaxClients 1500"
which is nonsense in any way as the default HARD_SERVER_LIMIT is far
smaller at 256). OpenBSD has, as the other
mentioned systems here besides linux, some compile-time ressource limits.
Especially counting here: OPEN_MAX and CLIENT_MAX. On linux 2.4 they are
changeable at runtime, and I guess the defaults are much higher than on
OpenBSD (duh! this must be the explanation for OpenBSD being that bad. all
ressource limits are quite low on OpenBSD per default - and this is a good
thing -, while they are quite high on linux at least). For production
webservers with serious load you want to change these in
/usr/src/sys/sys/syslimits.h (some sole soul told henning this is
configurable by login.conf(5) now, no need to change the source any more...
duh.); you want to use ulimit and you want to up NKMEMCLUSTERS and 
NMBCLUSTERS in kernel config. 
I don't have DDB, GPROF, KTRACE and KMEMSTATS in production kernels. Also
a statically linked apache is really faster than one using loadable modules
as the default one does. Pay special attention to your httpd binary, the
smaller it is the better. Also your httpd.conf should set MinSpareServers
and especially MaxSpareServer to _much_ higher values. Your are forcing
apache to kill clients and fork new ones with these small numbers all the
time. For production usage the KeepAlive settings are very important, too.
You save power on the server switching KeepAlive off, but this slows
down the access for each client alone. Setting KeepAliveTimeout to a slighly
smaller value is a good idea. This should not affect this performance test
(and shows once more why these are not that important for real operations).
For a fair comparision with linux you must also think of the mount method.
linux mounts its partitions async per default, with is a no-no on production
systems. OpenBSD mounts them "normal", enforcing metadata to be written
quite often, which is a good thing. ever seen a destroyed ffs partition
after a crash/power failure/whatever? I haven't. Seen destroyed ext2fs
partitions that were mounted async after these? Yes. Linux does not know a
"normal" mount, unfortunately, just "async" (fast, but unreliable) and
"sync" (reliable, but really slow). OpenBSD knows a sync mount too - I never
used it, but I guess OpenBSD in sync mode is far more paranoid and doing 
far more disk accesses that linux in sync mode - but that's just a guess.
Notheless harddisk performance must not be an issue on usual webservers (big
database backends may change this but should not dit on the same machine),
otherwise the server has not enough memory.
The network cards are really important. Never believed it until I saw the
difference myself. I'm a big fan of DECs 21143, revision 2/3. This chip and
a good PHY (I like the ICS1892 best, followed by the Seeq 84220; but this
may be influenced by the fact that I have coded OpenBSD support for these
two) gives a wonderful card.
http://www.holland-consulting.net/tech/ocep/21143.html is a great site about
these.
As a sidenote: it is a pity that the dc(4) driver in OpenBSD and above
is still broken, at least with 21143s and more than one card. Nobody willing
to help me with PR1859? (This is now solved in -current)
Oh, newer intel cards - fxp(4) - are reported to be good, too. No wonder,
they buyed DEC's network design team.
I don't know about the AIC7890 in your machine, there may be issues, too - I
really don't know, but I don't think so. Most of our servers run of simple
IDE disks as disk bandwidth is no issue on them. The mailservers use fast
dedicated SCSI disks for their queues, though. (Well, newer high quality IDE
drives such as IBM's 7200rpm ones are really fast and reliable, too).
196 MB RAM is not enough for a busy webserver, especially not if you are
using a GENERIC kernel and - more important - a default httpd. I use 512 MB
usually.
Another point: linux uses a dynamic part of its RAM for disk buffering, on
OpenBSD it is a fix value. This should be no issue, though, as - I said it
earlier - disk i/o must not be an issue on a webserver. RAM is important.
RAM RAM RAM. Save lots of money with IDE instead of SCSI (and AMD
Athlons/Durons instead of Intel stuff) and spend it for a high quality
Mainboard and especially High Quality and enough RAM. High Quality RAM means
Infineon modules here, nothing else. really. Not even foreign modules
equipped with Infineon chips. The SPD (the small EEPROM on SDRAM modules) is
_very_ important nowadays for Motherboards/BIOSses to use the correct
timing. The Infineon modules are the only one passing all test, fulfilling
all specs ok. Not only the SPD programming but all. 
Oh, finally, depending on how many parallel httpds you are seeing usually
(or while benchmarking here) you must adjust HARD_SERVER_LIMIT in apache's
src/include/httpd.h and MaxClients in httpd.conf. HARD_SERVER_LIMIT must be
higher than MaxClients, I usually use HARD_SERVER_LIMIT=2*MaxClients. For
OpenBSD, CLIENT_MAX should be around MaxClients*2 + 100, and OPEN_MAX
heavily depends on many other settings and your usage profile, MaxClients *
2 + 200 or so is a good idea. Note that both OPEN_MAX and CLIENT_MAX must be
less than 4000 if memory serves me right.
Won't affect performance too much, but a good idea on production systems:
turn off _all_ unneeded services. especially inetd.

I'm sure I forgot some small tweaks I've done on our servers, but I do
install them with a inhouse OpenBSD variant already tuned and stripped
(strippedOpenBSD, which is the public available variant of our inhouse
bswsOpenBSD, is on wiretapped.net). I also have internal scripts for setting
up an already tuned Apache and scripts to generate Apache's config out of
our backend databases (no, both not public available), so it is quite
possible that I forgot some tweaks here.

> (where is my valium?)

Gimme a bit, too, if you get some; maybe this helps me in getting some sleep
and stopping coding a few hours ;-))

> > > > Henning, are you reading this? We need your comments! :-)
> > > Yes, but John failed to give any details. His problem seems in no way
> > > related to OpenBSD or Apache config.
> > As I stated in my original posting:

As written in private mail: I'm really busy today and didn't visit this page
the first time. A dmesg included in the post would have been a good idea to
speed up things for the reader, too. Otherwise the site is a good idea for
giving as much details as possible to us.

> > > But you're right, I really should set up a webpage explaining using OpenBSD
> > > + Apache at serious load ;-)) 
> > That would be fantastic!

I think I'll just copy this mail somewhere on a webserver ;-))

> > At this point I'm left with the suspicion that the default install of
> > OpenBSD has "something" wrong with it. 

For normal Apache usage: no. For serious load one must tune any OS. The per
default small ressource limits on OpenBSD are a good thing; ressource limits
are important pieces in the security puzzle. Ever wondered why every
djb-daemon runs under softlimit and tcpserver always has a concurrency limit? 

> > Perhaps it just might be something
> > specific to the particular hardware I'm using not working well with
> > OpenBSD?

That's still possible. The numbers you get seem to be too low at a first
glance.

Greetings

Henning

-- 
* Henning Brauer, hostmaster@bsws.de, http://www.bsws.de *
* Roedingsmarkt 14, 20459 Hamburg, Germany               *
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)