 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
        |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Home / Documentation / User Manual 
Installation
Advanced Options
Protocols that Don't Load Balance Traffic Graphs Monthly client logs Security OS-specific Event Based I/O Operating System Considerations and Tuning
InstallationProto Balance can be installed by running the installation executable for your operating system downloaded from http://www.protonet.co.za/ This program must be run as superuser on Unix systems, and with admin privileges on Windows. On Windows you may get an alert "Windows firewall has blocked some features of this program". In this case, click on "Unblock". On Unix you must first log in as the "root" user. Proto Balance comes as an installation binary "install.bin" or "install.exe". Simply run the binary and follow the instructions. See below for Mac specific instructions.
QuickStart Proto Balance
All Unix systemsIf it does not already exist, create the directory where Proto Balance can store data and configuration files:
Note that if you use Proto Balance with the mmap option (see below) you will need 500MB of free disk space under this directory. The simplest way to invoke Proto Balance is with a shell command as follows:
Use the TERM signal (i.e. 15) to shutdown Proto Balance. Please go to the section "System Settings for High Performance" if you intend running Proto Balance with a large number of concurrent connections.
Microsoft WindowsOn Windows it is best to start Proto Balance from your Start menu. If you are an advanced system administrator you may want to use Proto Balance from the command-line. In this case read on: A directory would have already been created for Proto Balance to store data and configuration files. On Windows XP this directory is as follows. (Check by right clicking on "Proto Balance" in the Start menu and selecting "Properties" -- see "Start in:"). This is usually:
Invoke Proto Balance at the command prompt as follows:
Please go to the section "System Settings for High Performance" if you intend running Proto Balance with a large number of concurrent connections.
Apple Mac OS XProto Balance does not install itself like a Mac desktop application so it will not appear in any of your desktop menus. To install Proto Balance goto "Utilities" - "Terminal". Inside the Terminal window type,
then enter the system administrator password for your machine. You can now run the install executable depending on where you downloaded the file "install.bin". For example, if I am logged in as user "sally", I can enter:
Followed by:
Continue with the instructions for Unix above.
Ensuring that Proto Balance restarts after your machine has rebootedOn Unix, your operating system documentation should explain how to add the "protobalance" command to your start-up scripts. You will need to have system administrator privileges to make this change however. Use the "-daemon" and "-logfile" options to make Proto Balance run in the background and write to a log file. On Windows go to "Scheduled Tasks" in your "Control Panel" and click on the "Add Scheduled Task" wizard. Select "Proto Balance" from the list of Applications and then select the "When my computer starts" radio button as follows:
The remaining wizard options can be left as default. On Mac OS X, edit the file /etc/rc.local or create it if it does not exist. Give it execute permissions with the command "chmod a+x /etc/rc.local". This file should contain something like:
Proto Balance command-line options and flagsProto Balance displays a comprehensive explanation of command-line options when invoked with the -help option. Please run
to view options available on your operating system.
ConfigurationProto Balance's usefulness comes from its ease of configuration. This section will explain how to set up a simple cluster. To start, point your web browser to the URL where Proto Balance is running. If Proto Balance is running on the local machine, this URL will be http://127.0.0.1:8080/ If you have started Proto Balance with the -configport option, a port of other than 8080 must be specified in the URL. If you are configuring Proto Balance from a remote machine, its IP address must replace "127.0.0.1" in the URL. (Any web browser will work. The Proto Balance configuration web page does not use JavaScript.)
Logging in and setting your passwordWhen you access the configuration page for the first time, the login password will be empty - hence just click on "Login" without entering a password. For security you should set your password after logging in for the first time - click on "Change Password" under the "Login" tab. Proto Balance will remember the IP address of the machine your web browser is running on. If you access the configuration web page from a different machine, you will be forced to log in again.
Creating a forwarder to divert TCP connections to another hostThe simplest configuration is where Proto Balance listens on a port and forwards all data to a port on a different host. For instance, you can run Proto Balance on a firewall and forward all incoming mail to a mail server behind the firewall as follows: 1. Create a new cluster by clicking on "Add Cluster". 2. Enter the settings as follows. Replace "196.43.34.190" with the address of your outside (i.e. public) network interface. Note that "Unique Identifier" is a free text string that should briefly explain your cluster's purpose.
Clicking on the "Clusters" page will show your current settings:
3. Now add a physical mail server to which incoming connections will be forwarded. Click on "Add box":
Clicking on "Clusters" will now give:
Finally, click on the green traffic light to enable traffic to the physical machine (box). Your mail server will now receive mail.
Explanation of traffic lightsThe traffic lights are a unique feature of Proto Balance. At any time, it is easy to see the state of all physical machines (boxes) in the cluster. A square border around a light indicates that the light is active. The traffic lights have the following meaning:
Note that in this example it is useful to click on "MAIL MACHINE A" to see the "Last fallover" reason.
Creating a clusterBegin by creating the port forwarder example from the section "Creating a forwarder..." above. Now add a second box by clicking on "Add box" again:
Your clusters page will now look like:
Don't forget to enable your new box by clicking on the green traffic light. Also remember to click on "Save Config" so that your configuration persists across reboots. The above cluster will load balance between the mail machines 192.168.1.9 and 192.168.1.10. I.e. each new connection will be forwarded to the machine with the current lowest number of total connections so that the two machines evenly share the incoming mail load. It is easy to add additional machines to the cluster - even while the cluster is running.
Removing a box from a clusterYou can remove a box from the cluster even while Proto Balance is in full operation. To remove a box, carry out the following steps:
Connection OptionsThe options "Box connect timeout in milliseconds" and "Maximum number of connections", under the "Add Cluster" page, will be discussed in this section. TCP connections have a default "connect timeout" of 180 seconds. This means that if a remote client is trying to connect to a Proto Balance cluster through to a machine that has become disconnected from the physical network, Proto Balance will only give up trying the machine after 3 minutes. The client will wait unhappily for 3 full minutes before Proto Balance diverts it to an alternative machine in the cluster. In practice, Proto Balance should sensibly assume -- even after a few seconds -- that the machine is down if no response comes. This is because cluster network connections are high-speed connections that ordinarily respond within a few milliseconds (under moderate loading conditions). Settings for "Box connect timeout in milliseconds" might be in the range of 500 to 20000, with the high end of this range (20000) being the more conservative and safer setting. The setting "Maximum number of connections" instructs Proto Balance to drop new client connections when the total number of connections exceeds this setting. This option is useful for preventing Proto Balance from consuming resources that other applications might need, as well as to prevent your cluster from being overwhelmed. Most operating systems become unstable under this kind of extremely high load, hence you should set this value to the maximum number of connections that you are sure your cluster can cope with.
Advanced OptionsThis section discusses the settings under "Show advanced options" of the "Add Cluster" page, as well as further settings of individual machines in the cluster.
Box RetryNormally when a box is unavailable, Proto Balance shows a broken-connection icon (see "Explanation of traffic lights" above.) until such time as you might re-enable the box by clicking on the green traffic light. The following setting causes the box to revert to "green" state after a given number of seconds; in this example 30 seconds:
This means that, should the box fail, then new connections will be routed to an another box for 30 seconds, and the failed box will show the broken-connection icon. After 30 seconds, the box will automatically revert to "green" state. Should the box still be unavailable, the first new connection that discovers this will transparently be routed to another box in a way that is invisible to the client, and the box will show the broken connection icon as usual. If you use this option you should also set the option "Box connect timeout in milliseconds" to a low value because it may take this long to discover a failure - latency that may be unacceptable to your client application and may occur every 30 seconds. Should there be no new client connections after 30 seconds, the box will stay in "green" state indefinitely. This can be confusing as it would look like the box is working perfectly when in fact it is unavailable. Therefore you should always check your box's traffic graphs to see whether it really is working.
Load Balancing AlgorithmsThere are several ways to distribute clients across a cluster to ensure that all boxes in a cluster are evenly loaded. Proto Balance supports all commonly used load balancing algorithms. The possible settings are:
Connection rate limits, denial of service limits(Proto Balance Advanced and Proto Balance SSL only) The cluster setting "Maximum number of per-client connections per ten second period" works as follows: For each new successive connection attempt from a client, the client's new statistic is checked against that in the current ten second period. If the client would exceed the allotted quota, the connection is dropped. A line is also logged of the form "max connections of NNN per ten seconds reached for client NN.NN.NN.NN - dropping". If the client has not exceeded the allotted quota, the stats are incremented by one. If the ten-second window has rolled over, it is reset back to one. Similarly, the cluster setting "Maximum number of per-client concurrent connections" will set the quota on the total number of simultaneous connections a single client can have open. Connections that would exceed this limit are dropped, and a line logged of the form "max connections of NNN per ip reached for client NN.NN.NN.NN - dropping".
Idle TimeoutProto Balance Advanced and Proto Balance SSL have a feature to forcefully close connections that have been idle for too long. These would be connections over which no traffic has been transferred in either direction for some time. Set the configuration option "Client idle timeout in milliseconds" to a value in seconds. A value of zero disables this feature.
X-Forwarded-For HTTP headerProto Balance effectively hides the IP address of the connecting client. This is because, from your web server's point of view, the connection is coming from Proto Balance and not from a remote client. To log information about the connecting client, Proto Balance Advanced and Proto Balance SSL have the additional configuration setting, "Add X-Forwarded-For to HTTP header". Setting this to "Enabled" will harmlessly insert the IP address of the client into the request header sent to your web server. You will need your web server to log this field. In Apache this is easy: edit the file httpd.conf and change your log format to include the field,
For example, the www.protonet.co.za web site has the common log format as:
Note that there is a slight performance penalty when you do this because Proto Balance Advanced/SSL has to analyze the client request content. Protocols other than HTTP do not have the facility to add such headers.
Box affinityThere are some cluster environments where you would prefer a client to connect to the same physical box each time. This might be because each box caches local client data, whereas it is expensive to migrate that data to a sister box. For example, if a client happens to be directed to box 13 of 20 on first login, then the same client should also be directed to box 13 on all subsequent connection attempts. With Proto Balance there are two ways of doing this. The first method requires the following setting:
This setting works as follows: Consider a client connecting from the IP address 41.241.74.39. Proto Balance performs a cryptographic hash of the IP address modulus the number of boxes (See also "Box speed factor"). The result of this computation is that if there are, say, 20 boxes then 41.241.74.39 will map to the number 13 and the client will be directed to box 13. The same IP address will always map to the number 13 hence this client will continue to connect to box 13 as long has its IP address stays 41.241.74.39. Because the cryptographic hash is pseudo-random, load will be evenly spread over all boxes in the cluster. The second method requires the following setting:
This setting works as follows: Consider a client connecting for the first time. The client has an IP address of 41.241.74.39. A box, say box 13, is chosen for a client via one of the load-balancing algorithms discussed above. Proto Balance saves into a lookup table the client's IP address and chosen box. The next time the client connects, Proto Balance will find its IP address in the lookup table and then direct it to the same box as before.
Client connection dictionaryTo view the current state of the lookup table discussed in "Box affinity" above, click on "Show client connection dictionary" in the "Info" tab. This list shows five fields:
With Proto Balance Advanced and Proto Balance SSL, the dictionary has an upper limit on the number of clients after which records of the oldest clients are deleted. This limit can be set in the configuration web page - see "Max dictionary size in megabytes" under the Info tab. Note that each client consumes approximately 160 bytes therefore the default setting of 40 (megabytes) is sufficient to hold 250000 IP addresses. To enlarge this limit, increase your shared memory settings. With Proto Balance Advanced and Proto Balance SSL, the dictionary entries also have an expiry time. The setting "Delete an IP address from the dictionary after N seconds of inactivity" will set the upper age of a dictionary entry that has had no further connections from the same client IP address. The default setting of 604800 seconds equals seven days.
Box falloverAn option "Fallover Box" is useful in conjunction with box affinity explained above. This specifies the preferred box to route clients to in the event a box is unavailable. For instance if you have setup your cluster such that each box has a twin that contains replicas of its clients' data, then that twin would be the preferred box to send the client and twins would have each others' names in their "Fallover Box" setting. Here is an example:
Speed factorA clusters may have machines of differing CPU speed or other performance factor. For example, you may have a cluster made up both of machines of 500MHz CPUs and as well as machines of 750MHz CPUs. If you would like to direct proportionately more connections to the faster machines. You can set the speed factor to "2" for the 500Mhz machines and to "3" for the 750Mhz machines. You should use small ranges for this setting - i.e. do not use speed factor values of "100" or "200" when "1" or "2" is sufficient.
Close connection if...Proto Balance supports an obscure feature useful for IIOP connections. IIOP is the TCP protocol used by CORBA. If you are unsure what these terms mean then feel free to skip this section. There are many vendors that implement CORBA/IIOP: Borland(Visibroker/Janeva), BEA(WebLogic), Orbix, among others. There are some interoperability problems between certain versions of these implementations. Namely, some implementations don't respond to a CORBA "CloseConnection" request resulting in your application server being overwhelmed with connections. Proto Balance gives you the opportunity to force closing of a connection by setting this option as follows:
In this case Proto Balance will forcibly shutdown both the client and server side of the connection whenever it finds a packet matching the above binary sequence. Note that the above sequence is just an example - you should use a packet sniffer like http://www.ethereal.com/ to grab an example packet from your server and verify the problem - the packet can differ slightly depending on the application server architecture. Also be aware that this kind of forced shutdown introduces a race condition, albeit one that is rarely noticeable.
Multiple ClustersProto Balance allows you to create an arbitrary number of clusters, with an arbitrary number of boxes per cluster. The question is often asked: "How do I listen on more than one port?" This can easiliy be done by creating multiple "clusters" each listening on a different port. Note that all clusters share the same client connection dictionary (lookup table) discussed above. Hence if you are using the "remembering the client IP4/IP6 address" option in, say, two clusters, and it is foreseeable that the same client could connect to both of these clusters, then this is potentially a problem. You can avoid confusion by setting up both clusters with the same number of boxes having the same box names. As follows:
Alternatively, you can run multiple instances Proto Balance each on a different machine. (Note that under no circumstances should you ever try to startup more than one Proto Balance program on the same machine using the "-shared" option.) In general, you should not create more than 50 clusters, since this can degrade performance.
Protocols that Don't Load BalanceMost protocols can be proxied/redirected through Proto Balance and will work well. Examples of such protocols are those shown in the above examples, such as HTTP, SMTP, POP, etc. However there are protocols that encode TCP port and IP information into their data stream: To the server it appears as the though the connection is originating from Proto Balance whereas the data itself says that the connection is originating from a remote client. Load balancing such a protocol will not work. Examples of non-working protocols are BEA WebLogic "t3://" connections (with HTTP tunnelling disabled) and FTP connections. For the case of BEA WebLogic "t3://..." connections, change your client's connection setting to "http://..." instead of "t3://...", and check "Enable Tunnelling" on your WebLogic console as shown here:
BEA has more information about "HTTP tunneling" in their documentation. Another way to load balance t3:// connections is to use the same listening port as the WebLogic server (albeit on a different interface or different machine). This works because the t3:// protocol only encodes the port number and not the client address information. This means that, if your WebLogic server listens on port 7000, then all other WebLogic servers in your cluster must also listen on port 7000, and Proto Balance too must listen on port 7000.
Traffic GraphsProto Balance provides comprehensive traffic graphs which show failure histories as well as input and output traffic. Each box also has a thumbnail graph which shows traffic processed in the past minute:
If a box fails, then red tokens will appear on the graph. The following box had five pending connections at the time of failure; therefore the graph shows five tokens (even though box failed only once):
Clicking on the thumbnail will take you to a set of three graphs, each showing traffic over a different time scale:
Note that green represents bytes downloaded from your box to the client, whereas blue represents bytes uploaded from the client to your box. The above traffic is static HTTP content, hence there is little uploaded traffic compared to downloaded traffic. In this example the box was booted at 9:30pm. Maintenance was performed on the box at approximate 5am for about 10 minutes.
Monthly client logsProto Balance can create logs of all the clients that connect, to log files named according to the current year and month. The command-line option
specifies a folder where these logs are to be written, which folder must previously have been created and have the necessary access writes to allow Proto Balance to create log files. Log file names have the format YYYY-MM for example,
will create the files,
for the months September through November 2008. Proto Balance does not delete log files, therefore you will have to create a means of ensuring disk space is not used up. Example log lines are:
The figures for upload/download are in bytes. Each and every connection attempt is logged. If the X-Forwarded-For option (see the section X-Forwarded-For HTTP header) is enabled, then HTTP access logs will be included, for example:
SecurityThe Proto Balance configuration web page is not secure. You should use the command-line option -listeninterface to force Proto Balance to accept connections from only one interface. The interface you choose should be one that is not accessible from untrusted machines. For example, if you run Proto Balance on an Internet firewall, then -listeninterface should be the interface of your internal network. Proto Balance can improve the security of exposed networks when used to restrict access to only those services that Proto Balance is responsible for. For example, you may configure your firewall to forward no incoming Internet traffic, and at the same time configure Proto Balance to forward specific services to particular internal machines. Thus Proto Balance behaves like a discrete set of firewalling rules.
OS-specific Event Based I/O(This section is technical background.) Proto Balance Mach is able to use a high performance "event-based" I/O model specific to your particular operating system. An industry standard for high performance event-based I/O does not exist, and therefore each OS vendor has developed their own. The list is as follows:
Proto Balance supports all of these interfaces. The advantage of these interfaces over traditional programming models, is that Proto Balance can scale to arbitrarily large numbers of concurrent connections without incurring performance degradation. To disable "event-based" I/O, use the command-line option "-poll-method select" option. Consult the operating system specific sections below for more about "-poll-method" under "System Settings for High Performance with Your Operating System".
Operating System Considerations and TuningProto Balance uses a unique software architecture to be able to cope with so many concurrent connections while using as little operating system (OS) resources as possible. Little or no OS configuration is required. However, for enterprise-level environments where a high number of concurrent connections are expected, you should be aware of possible problems caused by limitations in your operating system. Proto Balance creates as many OS processes or threads as there are the-square-root-of the number of connections. This means that the number of connections handled by one OS process or thread is the square root of the total number of connections. Hence for many system resources, Proto Balance raises the OS limit by a power of 2. This gives Proto Balance exceptional scalability on any platform. For the discussions below, note that each client consumes two connections and hence two socket handles: one from the client, and one to the box.
Summary of limitsThe following table shows considerations for each of the three methods of invocation:
Note that for large numbers of concurrent connections, your operating system's settings will most certainly need customization. See the section "System Settings for High Performance..." below.
Maximum number of socket handles / file descriptorsOperating systems have a limit on the number of socket handles ("file descriptors" in Unix terminology) that a single process or thread can have open. Most OSs will have a means to increase these limits - consult your operating system (OS) documentation. Potential limits are:
Maximum number of threads (-pthread option)On some OSs, the Linux kernel in particular, there is a default limit on the maximum number of threads that a process can create. This means that when using the -pthread option, trying to create more that 200 (approx) clusters will result in a fatal error "pthread_create(): Cannot allocate memory". This limitation will usually come into effect when there are a large number of clusters with a large number of connections each. Should you have few clusters (say 4) and should you expect less than, say, 500 concurrent clients, then this limit will never be reached. Further, the per-process file descriptor limit will usually be reached first.
Shared memory and Semaphore limits (-shared shmat option)When Proto Balance is invoked with the option "-shared shmat" it starts multiple OS processes which communicate with each other using shared memory and semaphores (technically known as "System V IPC"). Some Unix operating systems have overly conservative shared memory limits - even modern ones. Because it can sometimes require a kernel rebuild to enlarge these limits, you may instead want to use the "-shared mmap" option - this will always work on all operating systems. Unfortunately, the option "-shared mmap" creates a cumbersome 500MB data swap file. Semaphore limits will come into affect with either -shared option, but are quite rare. If there is a problem with your settings, then you may get the error "shmat(): Too many open files", "shmat(): Out of memory" or "shmget(nnnnnnn): Invalid argument" and Proto Balance will abort. This can be catastrophic. Errors of the form "semop(): No space left on device" or other "sem.." errors similarly indicate a lack of semaphore resources. Shared memory has maximum limits imposed by the operating system, and a process may not exceed these. These limits are in terms of the allowed (a) number and (b) size of memory blocks, both (c) system-wide and (d) per-process. The same applies to limits on the number of semaphores. Proto Balance creates a small number of successively larger shared memory blocks. You must tune your OS to allow at least: 32 blocks per process, 1024 blocks system-wide and with a maximum block size of 128 megabytes (134217728 bytes) or more. All semaphore limits similarly need increasing to 128 through 256 per process. Each operating system has a different means of setting these limits. Consult the section "System Settings for High Performance..." below. Note that FreeBSD has a system resource PMAP_SHPGPERPROC which may need to be increased (pertaining to certain "collecting PV entries" error messages). A web search on "PMAP_SHPGPERPROC" will point you to the correct help.
TCP port limitsThis section is very important if you intend to handle high rates of new connections. All operating systems allow a maximum of 65535 TCP ports per network interface (IP address). Each connection requires a unique port. Hence a physical machine can only handle 65535 connections times the number of network interfaces. (The number of network interfaces is usually 1 or 2.) This limit is further reduced as will be explained. TCP does not dispose of the port once the connection has died - it holds the port for a considerable period of time - this is called the TIME-WAIT state of TCP. The TIME-WAIT state is a precautionary measure: the remote machine may have lost track of the state of the connection because of dropped packets, and may demand a response even when the connection is supposed to have ended. Officially, the TIME-WAIT state is supposed to last for four minutes. Now consider that most operating systems do not use the full range of ports - artificially limiting port allocation to, say, 40000 through 65000. Hence, if an application sustains a high rate of new connections, then OS port allocation may top-out at, say, 100 new connections per second - an unacceptably low rate. There are several TCP parameters that are always tunable on all operating systems that solve these problems:
Then there are various timeouts, adjustable on some operating systems but not on others.
Limits of Physical MemoryProto Balance consumes kernel and user resources of 20 megabytes of physical memory per 1000 concurrent connections. This figure varies widely for different operating systems. If you intend to handle 10'000 concurrent connections, then 512 to 1024 megabytes is recommended.
Other limitsEach OS has its own set of arbitrary limits when it comes to running high-performance software. The section "System Settings for High Performance..." contains more information.
System Settings for High Performance with Your Operating SystemProto Balance is designed to work efficiently with a large number of concurrent connections. With correct OS tuning Proto Balance can cope well with 10'000 concurrent connections. The operating system's effectiveness toward 10'000 concurrent connections depends on the hardware and the OS version. Proto Balance has by no means been tested to these extremes on every version of every OS, however where particular problems were encountered, these have been documented in the sections below. If you are not in an environment of thousands of concurrent connections, with hundreds of new connections per second, we recommend you do not adjust any TCP settings. Other settings always apply however. An attempt has been made to gather all the settings in one place for each supported OS. Please use these settings with care since the performance of other applications may be negatively impacted. Many of these settings are liable to render your system unstable, insecure, and/or unbootable if not done with the proper expertise. You are advised to follow your OS vendor's recommendations rather than any setting suggested in this document.
WindowsGoto your "Start" menu and select "Run...". Enter "regedit" as follows:
Navigate through the levels:
Right click on the white area to create a new "DWORD" registry entry "MaxFreeTcbs":
Double click on "MaxFreeTcbs" and select Decimal text, then enter "16000":
For Windows XP there are 6 new registry entries you must create in the same way:
If you are using Windows Server some of these entries do not apply. For Windows Server 2003 the default TcpNumConnections is 262144 and therefore need not be set. For Windows Server 2003 the values are:
If you are using Windows Vista and Windows Server 2008 the values are:
When you are done creating the registry entries, restart Windows. In addition, Windows Vista and Windows Server 2008 do not seem to be able to use the full range of ports without the "netsh" command executed under the Command Prompt. See http://support.microsoft.com/kb/929851/ for more information if you are using Proto Balance in conditions of high connection rates. Note that Windows XP prior to service pack 2 will not handle more than 3500 concurrent connections. When using "-poll-method iocp", Windows will use an "IO Completion Port" architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", Windows performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "iocp" is unsupported in that version and "select" is the default.) The following graph shows performance for both architectures (hardware is a commodity two core CPU at 1.2GHz):
LinuxLogin as root. Add these commands early in your system startup scripts:
(Note the setting for tcp_fin_timeout which has changed as of the present writing.) When using "-poll-method epoll", Linux will use an "epoll" event-based architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", Linux performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "epoll" is unsupported in that version and "select" is the default.) The following graph shows performance for both architectures (hardware is a commodity two core CPU at 1.2GHz):
MacOSLogin as root. Place the following lines into the file /etc/sysctl.conf (create it if it does not exist).
Otherwise run the following commands on startup:
MacOS has maximum limits on many of these values that cannot be overridden. Therefore you may not be able to create a large number of boxes or cluster. Yet in spite of this problem, MacOS 10.3 (Darwin kernel version 7.0.0) successfully starts 10'000 concurrent connections. When using "-poll-method kqueue" (the default), MacOS will use an "kqueue" event-based architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", MacOS performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "kqueue" is unsupported in that version and "select" is the default.)
HP-UXLogin as root. Run the commands:
(Note the setting for tcp_fin_wait_2_timeout which has changed as of the present writing.) Start sam and goto Kernel Configuration -> Configurable Parameters:
The parameters you will need to set are:
You will then need to create and boot a new kernel. These options appear when you Exit the Configuration Parameters window:
(Note that an old HP-UX model 9000/778/B180L won't perform well with over 1000 concurrent connections. With 100 concurrent connections this machine can consume 30% of the saturation of a 100 megabit full-duplex Ethernet interface, while Proto Balance consumes 95% CPU.) When using "-poll-method dev_poll" (the default), HP-UX will use an "/dev/poll" event-based architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", HP-UX performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "kqueue" is unsupported in that version and "select" is the default.)
SolarisLogin as root. Run the commands:
(Note the setting for tcp_fin_wait_2_flush_interval which has changed as of the present writing.) On Solaris 9 edit the file /etc/system, and append the lines:
On Solaris 10 edit the file /etc/system, and append the lines:
Then reboot. When using "-poll-method dev_poll" (the default), Solaris will use an "/dev/poll" event-based architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", Solaris performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "kqueue" is unsupported in that version and "select" is the default.)
FreeBSDLogin as root. Edit the file /boot/loader.conf and append the lines:
For Proto Balance Mail, use:
Then reboot. It would seem that the above kern.ipc.shmmax=134217728 setting sometimes does not take effect, resulting in an "shmget(nnnnnnn): Invalid argument" error. Proto Balance then aborts. In this case you need to rebuild your kernel with a larger SYSPTSIZE setting, and a larger SHMALL and SHMSEG setting. Consult FreeBSD resources for information on how to do this. Be sure to use the latest stable FreeBSD release on 64-bit architecture. When using "-poll-method kqueue" (the default), FreeBSD will use an "kqueue" event-based architecture and will have near-perfectly scalability up to 10000 concurrent connections. When using "-poll-method select", FreeBSD performance degrades substantially with large numbers of concurrent connections. (If your version of Proto Balance does not support the option "-poll-method" it means "kqueue" is unsupported in that version and "select" is the default.) Frequently Asked Questions
Portions of this software: Copyright (c) 1990, 1993, 1994 The Regents of the University of California. All rights reserved. This code is derived from software contributed to Berkeley by Margo Seltzer. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Microsoft Windows
