NAME

lond - ``LON Daemon'' Server (port ``LOND'' 5663)


SYNOPSIS

Usage: lond

Should only be run as user=www. This is a command-line script which is invoked by loncron. There is no expectation that a typical user will manually start lond from the command-line. (In other words, DO NOT START lond YOURSELF.)


DESCRIPTION

There are two characteristics associated with the running of lond, PROCESS MANAGEMENT (starting, stopping, handling child processes) and SERVER-SIDE ACTIVITIES (password authentication, user creation, subscriptions, etc). These are described in two large sections below.

PROCESS MANAGEMENT

Preforker - server who forks first. Runs as a daemon. HUPs. Uses IDEA encryption

lond forks off children processes that correspond to the other servers in the network. Management of these processes can be done at the parent process level or the child process level.

logs/lond.log is the location of log messages.

The process management is now explained in terms of linux shell commands, subroutines internal to this code, and signal assignments:

SERVER-SIDE ACTIVITIES

Server-side information can be accepted in an encrypted or non-encrypted method.

ping

Query a client in the hosts.tab table; ``Are you there?''

pong

Respond to a ping query.

ekey

Read in encrypted key, make cipher. Respond with a buildkey.

load

Respond with CPU load based on a computation upon /proc/loadavg.

currentauth

Reply with current authentication information (only over an encrypted channel).

auth

Only over an encrypted channel, reply as to whether a user's authentication information can be validated.

passwd

Allow for a password to be set.

makeuser

Make a user.

passwd

Allow for authentication mechanism and password to be changed.

home

Respond to a question ``are you the home for a given user?''

update

Update contents of a subscribed resource.

unsubscribe

The server is unsubscribing from a resource.

subscribe

The server is subscribing to a resource.

log

Place in logs/lond.log

put

stores hash in namespace

rolesputy

put a role into a user's environment

get

returns hash with keys from array reference filled in from namespace

eget

returns hash with keys from array reference filled in from namesp (encrypts the return communication)

rolesget

get a role from a user's environment

del

deletes keys out of array from namespace

keys

returns namespace keys

dump

dumps the complete (or key matching regexp) namespace into a hash

store

stores hash permanently for this url; hashref needs to be given and should be a \%hashname; the remaining args aren't required and if they aren't passed or are '' they will be derived from the ENV

restore

returns a hash for a given url

querysend

Tells client about the lonsql process that has been launched in response to a sent query.

queryreply

Accept information from lonsql and make appropriate storage in temporary file space.

idput

Defines usernames as corresponding to IDs. (These ``IDs'' are unique identifiers for each student, defined perhaps by the institutional Registrar.)

idget

Returns usernames corresponding to IDs. (These ``IDs'' are unique identifiers for each student, defined perhaps by the institutional Registrar.)

tmpput

Accept and store information in temporary space.

tmpget

Send along temporarily stored information.

ls

List part of a user's directory.

pushtable

Pushes a file in /home/httpd/lonTab directory. Currently limited to: hosts.tab and domain.tab. The old file is copied to *.tab.backup but must be restored manually in case of a problem with the new table file. pushtable requires that the request be encrypted and validated via ValidateManager. The form of the command is: enc:pushtable tablename <tablecontents> \n where pushtable, tablename and <tablecontents> will be encrypted, but \n is a cleartext newline.

Hanging up (exit or init)

What to do when a client tells the server that they (the client) are leaving the network.

unknown command

If lond is sent an unknown command (not in the list above), it replys to the client ``unknown_cmd''.

UNKNOWN CLIENT

If the anti-spoofing algorithm cannot verify the client, the client is rejected (with a ``refused'' message sent to the client, and the connection is closed.


PREREQUISITES

IO::Socket IO::File Apache::File POSIX Crypt::IDEA LWP::UserAgent() GDBM_File Authen::Krb4 Authen::Krb5


COREQUISITES


OSNAMES

linux


SCRIPT CATEGORIES

Server/Process


LOG MESSAGES

The messages below can be emitted in the lond log. This log is located in ~httpd/perl/logs/lond.log Many log messages have HTML encapsulation to provide coloring if examined from inside a web page. Some do not. Where color is used, the colors are; Red for sometihhng to get excited about and to follow up on. Yellow for something to keep an eye on to be sure it does not get worse, Green,and Blue for informational items.

In the discussions below, sometimes reference is made to ~httpd when describing file locations. There isn't really an httpd user, however there is an httpd directory that gets installed in the place that user home directories go. On linux, this is usually (always?) /home/httpd.

Some messages are colorless. These are usually (not always) Green/Blue color level messages.

(Red) LocalConnection rejecting non local: <ip> ne 127.0.0.1

A local connection negotiation was attempted by a host whose IP address was not 127.0.0.1. The socket is closed and the child will exit. lond has three ways to establish an encyrption key with a client:

local

The key is written and read from a file. This is only valid for connections from localhost.

insecure

The key is generated by the server and transmitted to the client.

ssl (secure)

An ssl connection is negotiated with the client, the key is generated by the server and sent to the client across this ssl connection before the ssl connectionis terminated and clear text transmission resumes.

(Red) LocalConnection: caller is insane! init = <init> and type = <type>

The client is local but has not sent an initialization string that is the literal ``init:local'' The connection is closed and the child exits.

Red CRITICAL Can't get key file <error>

SSL key negotiation is being attempted but the call to lonssl::KeyFile failed. This usually means that the configuration file is not correctly defining or protecting the directories/files lonCertificateDirectory or lonnetPrivateKey <error> is a string that describes the reason that the key file could not be located.

(Red) CRITICAL Can't get certificates <error>

SSL key negotiation failed because we were not able to retrives our certificate or the CA's certificate in the call to lonssl::CertificateFile <error> is the textual reason this failed. Usual reasons:


=item Apache config file for loncapa  incorrect:

one of the variables 
lonCertificateDirectory, lonnetCertificateAuthority, or lonnetCertificate
undefined or incorrect
Permission error:

The directory pointed to by lonCertificateDirectory is not readable by lond

Permission error:

Files in the directory pointed to by lonCertificateDirectory are not readable by lond.

Installation error:

Either the certificate authority file or the certificate have not been installed in lonCertificateDirectory.

(Red) CRITICAL SSL Socket promotion failed: <err>

The promotion of the connection from plaintext to SSL failed <err> is the reason for the failure. There are two system calls involved in the promotion (one of which failed), a dup to produce a second fd on the raw socket over which the encrypted data will flow and IO::SOcket::SSL->new_from_fd which creates the SSL connection on the duped fd.

(Blue) WARNING client did not respond to challenge

This occurs on an insecure (non SSL) connection negotiation request. lond generates some number from the time, the PID and sends it to the client. The client must respond by echoing this information back. If the client does not do so, that's a violation of the challenge protocols and the connection will be failed.

(Red) No manager table. Nobody can manage!!

lond has the concept of privileged hosts that can perform remote management function such as update the hosts.tab. The manager hosts are described in the ~httpd/lonTabs/managers.tab file. this message is logged if this file is missing.

(Green) Registering manager <dnsname> as <cluster_name> with <ipaddress>

Reports the successful parse and registration of a specific manager.

Green existing host <clustername:dnsname>

The manager host is already defined in the hosts.tab the information in that table, rather than the info in the manager table will be used to determine the manager's ip.

(Red) Unable to craete <filename>

lond has been asked to create new versions of an administrative file (by a manager). When this is done, the new file is created in a temp file and then renamed into place so that there are always usable administrative files, even if the update fails. This failure message means that the temp file could not be created. The update is abandoned, and the old file is available for use.

(Green) CopyFile from <oldname> to <newname> failed

In an update of administrative files, the copy of the existing file to a backup file failed. The installation of the new file may still succeed, but there will not be a back up file to rever to (this should probably be yellow).

(Green) Pushfile: backed up <oldname> to <newname>

See above, the backup of the old administrative file succeeded.

(Red) Pushfile: Unable to install <filename> <reason>

The new administrative file could not be installed. In this case, the old administrative file is still in use.

(Green) Installed new < filename>.

The new administrative file was successfullly installed.

(Red) Reinitializing lond pid=<pid>

The lonc child process <pid> will be sent a USR2 signal.

(Red) Reinitializing self

We've been asked to re-read our administrative files,and are doing so.

(Yellow) error:Invalid process identifier <ident>

A reinit command was received, but the target part of the command was not valid. It must be either 'lond' or 'lonc' but was <ident>

(Green) isValideditCommand checking: Command = <command> Key = <key> newline = <newline>

Checking to see if lond has been handed a valid edit command. It is possible the edit command is not valid in that case there are no log messages to indicate that.

Result of password change for <username> pwchange_success

The password for <username> was successfully changed.

Unable to open <user> passwd to change password

Could not rewrite the internal password file for a user

Result of password change for <user> : <result>

A unix password change for <user> was attempted 
and the pipe returned <result>
LWP GET: <message> for <fname> (<remoteurl>)

The lightweight process fetch for a resource failed with <message> the local filename that should have existed/been created was <fname> the corresponding URI: <remoteurl> This is emitted in several places.

Unable to move <transname> to <destname>

From fetch_user_file_handler - the user file was replicated but could not be mv'd to its final location.

Looking for <domain> <username>

From user_has_session_handler - This should be a Debug call instead it indicates lond is about to check whether the specified user has a session active on the specified domain on the local host.

Client <ip> (<name>) hanging up: <input>

lond has been asked to exit by its client. The <ip> and <name> identify the client systemand <input> is the full exit command sent to the server.

Red CRITICAL: ABNORMAL EXIT. child <pid> for server <hostname> died through a crass with this error->[<message>].

A lond child terminated.  NOte that this termination can also occur when the
child receives the QUIT or DIE signals.  <pid> is the process id of the child,
<hostname> the host lond is working for, and <message> the reason the child died
to the best of our ability to get it (I would guess that any numeric value
represents and errno value).  This is immediately followed by
Famous last words: Catching exception - <log>

Where log is some recent information about the state of the child.

Red CRITICAL: TIME OUT <pid>

Some timeout occured for server <pid>. THis is normally a timeout on an LWP doing an HTTP::GET.

child <pid> died

The reaper caught a SIGCHILD for the lond child process <pid> This should be modified to also display the IP of the dying child $children{$pid}

Unknown child 0 died A child died but the wait for it returned a pid of zero which really should not ever happen.
Child <which> - <pid> looks like we missed it's death

When a sigchild is received, the reaper process checks all children to see if they are alive. If children are dying quite quickly, the lack of signal queuing can mean that a signal hearalds the death of more than one child. If so this message indicates which other one died. <which> is the ip of a dead child

Free socket: <shutdownretval>

The HUNTSMAN sub was called due to a SIGINT in a child process. The socket is being shutdown. for whatever reason, <shutdownretval> is printed but in fact shutdown() is not documented to return anything. This is followed by:

Red CRITICAL: Shutting down

Just prior to exit.

Free socket: <shutdownretval>

The HUPSMAN sub was called due to a SIGHUP. all children get killsed, and lond execs itself. This is followed by:

(Red) CRITICAL: Restarting

lond is about to exec itself to restart.

(Blue) Updating connections

(In response to a USR2). All the children (except the one for localhost) are about to be killed, the hosts tab reread, and Apache reloaded via apachereload.

(Blue) UpdateHosts killing child <pid> for ip <ip>

Due to USR2 as above.

(Green) keeping child for ip <ip> (pid = <pid>)

In response to USR2 as above, the child indicated is not being restarted because it's assumed that we'll always need a child for the localhost.

Going to check on the children

Parent is about to check on the health of the child processes. Note that this is in response to a USR1 sent to the parent lond. there may be one or more of the next two messages:

<pid> is dead

A child that we have in our child hash as alive has evidently died.

Child <pid> did not respond

In the health check the child <pid> did not update/produce a pid_.txt file when sent it's USR1 signal. That process is killed with a 9 signal, as it's assumed to be hung in some un-fixable way.

Finished checking children

Master processs's USR1 processing is cojmplete.
(Red) CRITICAL: ------- Starting ------

(There are more '-'s on either side). Lond has forked itself off to form a new session and is about to start actual initialization.

(Green) Attempting to start child (<client>)

Started a new child process for <client>. Client is IO::Socket object connected to the child. This was as a result of a TCP/IP connection from a client.

Unable to determine who caller was, getpeername returned nothing In child process initialization. either getpeername returned undef or a zero sized object was returned. Processing continues, but in my opinion, this should be cause for the child to exit.
Unable to determine clientip

In child process initialization. The peer address from getpeername was not defined. The client address is stored as ``Unavailable'' and processing continues.

(Yellow) INFO: Connection <ip> <name> connection type = <type> In child initialization. A good connectionw as received from <ip>.
<name>

is the name of the client from hosts.tab.

<type>

Is the connection type which is either

manager

The connection is from a manager node, not in hosts.tab

client

the connection is from a non-manager in the hosts.tab

both

The connection is from a manager in the hosts.tab.

(Blue) Certificates not installed -- trying insecure auth

One of the certificate file, key file or certificate authority file could not be found for a client attempting SSL connection intiation. COnnection will be attemptied in in-secure mode. (this would be a system with an up to date lond that has not gotten a certificate from us).

(Green) Successful local authentication

A local connection successfully negotiated the encryption key. In this case the IDEA key is in a file (that is hopefully well protected).

(Green) Successful ssl authentication with <client>

The client (<client> is the peer's name in hosts.tab), has successfully negotiated an SSL connection with this child process.

(Green) Successful insecure authentication with <client>

The client has successfully negotiated an insecure connection withthe child process.

(Yellow) Attempted insecure connection disallowed

The client attempted and failed to successfully negotiate a successful insecure connection. This can happen either because the variable londAllowInsecure is false or undefined, or becuse the child did not successfully echo back the challenge string.