lond - ``LON Daemon'' Server (port ``LOND'' 5663)
Usage: lond
Should only be run as user=www. This is a command-line script which is invoked by loncron. There is no expectation that a typical user will manually start lond from the command-line. (In other words, DO NOT START lond YOURSELF.)
There are two characteristics associated with the running of lond, PROCESS MANAGEMENT (starting, stopping, handling child processes) and SERVER-SIDE ACTIVITIES (password authentication, user creation, subscriptions, etc). These are described in two large sections below.
PROCESS MANAGEMENT
Preforker - server who forks first. Runs as a daemon. HUPs. Uses IDEA encryption
lond forks off children processes that correspond to the other servers in the network. Management of these processes can be done at the parent process level or the child process level.
logs/lond.log is the location of log messages.
The process management is now explained in terms of linux shell commands, subroutines internal to this code, and signal assignments:
PID is stored in logs/lond.pid
This is the process id number of the parent lond process.
SIGTERM and SIGINT
Parent signal assignment: $SIG{INT} = $SIG{TERM} = \&HUNTSMAN;
Child signal assignment: $SIG{INT} = 'DEFAULT'; (and SIGTERM is DEFAULT also) (The child dies and a SIGALRM is sent to parent, awaking parent from slumber to restart a new child.)
Command-line invocations: kill -s SIGTERM PID kill -s SIGINT PID
Subroutine HUNTSMAN: This is only invoked for the lond parent PID. This kills all the children, and then the parent. The lonc.pid file is cleared.
SIGHUP
Current bug: This signal can only be processed the first time on the parent process. Subsequent SIGHUP signals have no effect.
Parent signal assignment: $SIG{HUP} = \&HUPSMAN;
Child signal assignment: none (nothing happens)
Command-line invocations: kill -s SIGHUP PID
Subroutine HUPSMAN: This is only invoked for the lond parent PID, This kills all the children, and then the parent. The lond.pid file is cleared.
SIGUSR1
Parent signal assignment: $SIG{USR1} = \&USRMAN;
Child signal assignment: $SIG{USR1}= \&logstatus;
Command-line invocations: kill -s SIGUSR1 PID
Subroutine USRMAN: When invoked for the lond parent PID, SIGUSR1 is sent to all the children, and the status of each connection is logged.
SIGUSR2
Parent Signal assignment: $SIG{USR2} = \&UpdateHosts
Child signal assignment: NONE
SIGCHLD
Parent signal assignment: $SIG{CHLD} = \&REAPER;
Child signal assignment: none
Command-line invocations: kill -s SIGCHLD PID
Subroutine REAPER: This is only invoked for the lond parent PID. Information pertaining to the child is removed. The socket port is cleaned up.
SERVER-SIDE ACTIVITIES
Server-side information can be accepted in an encrypted or non-encrypted method.
Query a client in the hosts.tab table; ``Are you there?''
Respond to a ping query.
Read in encrypted key, make cipher. Respond with a buildkey.
Respond with CPU load based on a computation upon /proc/loadavg.
Reply with current authentication information (only over an encrypted channel).
Only over an encrypted channel, reply as to whether a user's authentication information can be validated.
Allow for a password to be set.
Make a user.
Allow for authentication mechanism and password to be changed.
Respond to a question ``are you the home for a given user?''
Update contents of a subscribed resource.
The server is unsubscribing from a resource.
The server is subscribing to a resource.
Place in logs/lond.log
stores hash in namespace
put a role into a user's environment
returns hash with keys from array reference filled in from namespace
returns hash with keys from array reference filled in from namesp (encrypts the return communication)
get a role from a user's environment
deletes keys out of array from namespace
returns namespace keys
dumps the complete (or key matching regexp) namespace into a hash
stores hash permanently for this url; hashref needs to be given and should be a \%hashname; the remaining args aren't required and if they aren't passed or are '' they will be derived from the ENV
returns a hash for a given url
Tells client about the lonsql process that has been launched in response to a sent query.
Accept information from lonsql and make appropriate storage in temporary file space.
Defines usernames as corresponding to IDs. (These ``IDs'' are unique identifiers for each student, defined perhaps by the institutional Registrar.)
Returns usernames corresponding to IDs. (These ``IDs'' are unique identifiers for each student, defined perhaps by the institutional Registrar.)
Accept and store information in temporary space.
Send along temporarily stored information.
List part of a user's directory.
Pushes a file in /home/httpd/lonTab directory. Currently limited to: hosts.tab and domain.tab. The old file is copied to *.tab.backup but must be restored manually in case of a problem with the new table file. pushtable requires that the request be encrypted and validated via ValidateManager. The form of the command is: enc:pushtable tablename <tablecontents> \n where pushtable, tablename and <tablecontents> will be encrypted, but \n is a cleartext newline.
What to do when a client tells the server that they (the client) are leaving the network.
If lond is sent an unknown command (not in the list above), it replys to the client ``unknown_cmd''.
If the anti-spoofing algorithm cannot verify the client, the client is rejected (with a ``refused'' message sent to the client, and the connection is closed.
IO::Socket IO::File Apache::File POSIX Crypt::IDEA LWP::UserAgent() GDBM_File Authen::Krb4 Authen::Krb5
linux
Server/Process
The messages below can be emitted in the lond log. This log is located in ~httpd/perl/logs/lond.log Many log messages have HTML encapsulation to provide coloring if examined from inside a web page. Some do not. Where color is used, the colors are; Red for sometihhng to get excited about and to follow up on. Yellow for something to keep an eye on to be sure it does not get worse, Green,and Blue for informational items.
In the discussions below, sometimes reference is made to ~httpd when describing file locations. There isn't really an httpd user, however there is an httpd directory that gets installed in the place that user home directories go. On linux, this is usually (always?) /home/httpd.
Some messages are colorless. These are usually (not always) Green/Blue color level messages.
A local connection negotiation was attempted by a host whose IP address was not 127.0.0.1. The socket is closed and the child will exit. lond has three ways to establish an encyrption key with a client:
The key is written and read from a file. This is only valid for connections from localhost.
The key is generated by the server and transmitted to the client.
An ssl connection is negotiated with the client, the key is generated by the server and sent to the client across this ssl connection before the ssl connectionis terminated and clear text transmission resumes.
The client is local but has not sent an initialization string that is the literal ``init:local'' The connection is closed and the child exits.
SSL key negotiation is being attempted but the call to lonssl::KeyFile failed. This usually means that the configuration file is not correctly defining or protecting the directories/files lonCertificateDirectory or lonnetPrivateKey <error> is a string that describes the reason that the key file could not be located.
SSL key negotiation failed because we were not able to retrives our certificate or the CA's certificate in the call to lonssl::CertificateFile <error> is the textual reason this failed. Usual reasons:
=item Apache config file for loncapa incorrect:
one of the variables lonCertificateDirectory, lonnetCertificateAuthority, or lonnetCertificate undefined or incorrect
The directory pointed to by lonCertificateDirectory is not readable by lond
Files in the directory pointed to by lonCertificateDirectory are not readable by lond.
Either the certificate authority file or the certificate have not been installed in lonCertificateDirectory.
The promotion of the connection from plaintext to SSL failed <err> is the reason for the failure. There are two system calls involved in the promotion (one of which failed), a dup to produce a second fd on the raw socket over which the encrypted data will flow and IO::SOcket::SSL->new_from_fd which creates the SSL connection on the duped fd.
This occurs on an insecure (non SSL) connection negotiation request. lond generates some number from the time, the PID and sends it to the client. The client must respond by echoing this information back. If the client does not do so, that's a violation of the challenge protocols and the connection will be failed.
lond has the concept of privileged hosts that can perform remote management function such as update the hosts.tab. The manager hosts are described in the ~httpd/lonTabs/managers.tab file. this message is logged if this file is missing.
Reports the successful parse and registration of a specific manager.
The manager host is already defined in the hosts.tab the information in that table, rather than the info in the manager table will be used to determine the manager's ip.
lond has been asked to create new versions of an administrative file (by a manager). When this is done, the new file is created in a temp file and then renamed into place so that there are always usable administrative files, even if the update fails. This failure message means that the temp file could not be created. The update is abandoned, and the old file is available for use.
In an update of administrative files, the copy of the existing file to a backup file failed. The installation of the new file may still succeed, but there will not be a back up file to rever to (this should probably be yellow).
See above, the backup of the old administrative file succeeded.
The new administrative file could not be installed. In this case, the old administrative file is still in use.
The new administrative file was successfullly installed.
The lonc child process <pid> will be sent a USR2 signal.
We've been asked to re-read our administrative files,and are doing so.
A reinit command was received, but the target part of the command was not valid. It must be either 'lond' or 'lonc' but was <ident>
Checking to see if lond has been handed a valid edit command. It is possible the edit command is not valid in that case there are no log messages to indicate that.
The password for <username> was successfully changed.
Could not rewrite the internal password file for a user
A unix password change for <user> was attempted and the pipe returned <result>
The lightweight process fetch for a resource failed with <message> the local filename that should have existed/been created was <fname> the corresponding URI: <remoteurl> This is emitted in several places.
From fetch_user_file_handler - the user file was replicated but could not be mv'd to its final location.
From user_has_session_handler - This should be a Debug call instead it indicates lond is about to check whether the specified user has a session active on the specified domain on the local host.
lond has been asked to exit by its client. The <ip> and <name> identify the client systemand <input> is the full exit command sent to the server.
A lond child terminated. NOte that this termination can also occur when the child receives the QUIT or DIE signals. <pid> is the process id of the child, <hostname> the host lond is working for, and <message> the reason the child died to the best of our ability to get it (I would guess that any numeric value represents and errno value). This is immediately followed by
Where log is some recent information about the state of the child.
Some timeout occured for server <pid>. THis is normally a timeout on an LWP doing an HTTP::GET.
The reaper caught a SIGCHILD for the lond child process <pid> This should be modified to also display the IP of the dying child $children{$pid}
When a sigchild is received, the reaper process checks all children to see if they are alive. If children are dying quite quickly, the lack of signal queuing can mean that a signal hearalds the death of more than one child. If so this message indicates which other one died. <which> is the ip of a dead child
The HUNTSMAN sub was called due to a SIGINT in a child process. The socket is being shutdown.
for whatever reason, <shutdownretval> is printed but in fact shutdown()
is not documented
to return anything. This is followed by:
Just prior to exit.
The HUPSMAN sub was called due to a SIGHUP. all children get killsed, and lond execs itself. This is followed by:
lond is about to exec itself to restart.
(In response to a USR2). All the children (except the one for localhost) are about to be killed, the hosts tab reread, and Apache reloaded via apachereload.
Due to USR2 as above.
In response to USR2 as above, the child indicated is not being restarted because it's assumed that we'll always need a child for the localhost.
Parent is about to check on the health of the child processes. Note that this is in response to a USR1 sent to the parent lond. there may be one or more of the next two messages:
A child that we have in our child hash as alive has evidently died.
In the health check the child <pid> did not update/produce a pid_.txt file when sent it's USR1 signal. That process is killed with a 9 signal, as it's assumed to be hung in some un-fixable way.
Master processs's USR1 processing is cojmplete.
(There are more '-'s on either side). Lond has forked itself off to form a new session and is about to start actual initialization.
Started a new child process for <client>. Client is IO::Socket object connected to the child. This was as a result of a TCP/IP connection from a client.
In child process initialization. The peer address from getpeername was not defined. The client address is stored as ``Unavailable'' and processing continues.
is the name of the client from hosts.tab.
Is the connection type which is either
The connection is from a manager node, not in hosts.tab
the connection is from a non-manager in the hosts.tab
The connection is from a manager in the hosts.tab.
One of the certificate file, key file or certificate authority file could not be found for a client attempting SSL connection intiation. COnnection will be attemptied in in-secure mode. (this would be a system with an up to date lond that has not gotten a certificate from us).
A local connection successfully negotiated the encryption key. In this case the IDEA key is in a file (that is hopefully well protected).
The client (<client> is the peer's name in hosts.tab), has successfully negotiated an SSL connection with this child process.
The client has successfully negotiated an insecure connection withthe child process.
The client attempted and failed to successfully negotiate a successful insecure connection. This can happen either because the variable londAllowInsecure is false or undefined, or becuse the child did not successfully echo back the challenge string.