Many of us have experienced this problem: you have some process (for example a web server) running as a service. For a modern Linux distribution that implies it was started using systemctl. It runs under some designated user id and its output is sent to the system log infrastructure. You can look at this output using journalctl. What do you do if it misbehaves?

If you are lucky, the log output gives a clue or produces enough information to make an educated guess on how to trigger the same issue while running the same service interactively on your development machine. Then you can fix the software, upload it to the server machine and use systemctl restart myservice to activate the patched version. The only price paid is a short down time for the restart. The effect of this may vary from zero impact if there are redundant services to a serious down time if there is no redundancy and the start time is long.

How can we improve?

Well, SWI-Prolog can recompile modified sources safely into a running process. If the service was started with library(http/http_unix_daemon), the default is to handle SIGHUP to run make/0 to reload modified source files. We can use that for two purposes:

If the log files do not provide sufficient detail, add debug/3 statements to the code and run `systemctl reload myservice` to get more detailed information. Do not forget to activate the debug topic by adding this to the modified source file
```
:- debug(mytopic).
```
After we patched the sources locally or using the scenario above, use systemctl reload myservice to restart without downtime.

This may get clumsy though, in particular if we want to examine some state or run some query in the context of the service. It can be done by misusing directions as the above debug/1 call and capture the output in the log files. There must be something better, no?

Using SSH

The Prolog pack libssh allows you to create an SSH server in the Prolog process. Once that is installed we can login to the server at any time. We are protected by the de-facto standard and secure remote login protocol provided by SSH. As we login to the server the server starts a thread that provides a normal Prolog toplevel. If the (default) editline command line editor is compiled into the server we have a fully functional Prolog toplevel including command editing, completion, history and color output. This implies

We can easily examine global status such as global dynamic predicates, shared tables, etc.
We can run test queries against any predicate. Such calls are executed in the context of the life service. The full context of the server may be hard to reproduce on a development machine. Surely it is wise to setup your development and maintenance system to be able to reproduce this environment as precisely as possible. This may be costly or impossible though.
We can examine threads using threads/0 or tbacktrace/1.
We can kill a thread misbehaves using thread_signal/2 sending abort/0 as signal.
We can easily reload modified sources using make/0 doing the same as the above systemctl reload myservice in a bit more friendly way.
We can (de-)activate debug/3 statements by calling debug/1.
... and much more ...

Setting up ssh to allow login to a server

The first step to make all this magic possible is to install the libssh package. To do so, you first need to install the dependencies. On a Debian based Linux machine this implies:

apt install build-essential ssh-client libssh-dev

Now start Prolog and install the ssh pack by running ?- pack_install(libssh). Below is the output of the install session. Make sure there are no errors.

swipl
?- pack_install(libssh).
% Contacting server at https://www.swi-prolog.org/pack/query ... ok
Install libssh@0.9.1 from https://github.com/JanWielemaker/libssh/archive/V0.9.1.zip Y/n?
% Contacting server at https://www.swi-prolog.org/pack/query ... ok
% "V0.9.1.zip" was downloaded 16 times
Package:                libssh
Title:                  Provide an embedded SSH server
Installed version:      0.9.1
Author:                 Jan Wielemaker <jan@swi-prolog.org>
Home page:              https://github.com/JanWielemaker/libssh
Download URL:           https://github.com/JanWielemaker/libssh/archive/*.zip
Install "libssh-0.9.1.zip" (20,931 bytes) Y/n?
% -- The C compiler identification is GNU 10.2.0
% -- The CXX compiler identification is GNU 10.2.0
% -- Check for working C compiler: /usr/bin/X11/cc
% -- Check for working C compiler: /usr/bin/X11/cc -- works
% -- Detecting C compiler ABI info
% -- Detecting C compiler ABI info - done
% -- Detecting C compile features
% -- Detecting C compile features - done
% -- Check for working CXX compiler: /usr/bin/X11/c++
% -- Check for working CXX compiler: /usr/bin/X11/c++ -- works
% -- Detecting CXX compiler ABI info
% -- Detecting CXX compiler ABI info - done
% -- Detecting CXX compile features
% -- Detecting CXX compile features - done
% -- Found PkgConfig: /home/jan/bin/pkg-config (found version "0.29.2")
% -- Found LIBSWIPL: /usr/lib/libswipl.so
% -- Found LIBSSH: /usr/lib/x86_64-linux-gnu/libssh.so
% -- Looking for include file pty.h
% -- Looking for include file pty.h - found
% -- Looking for include file util.h
% -- Looking for include file util.h - not found
% -- Looking for gettid
% -- Looking for gettid - found
% -- Configuring done
% -- Generating done
% -- Build files have been written to: /home/jan/.local/share/swi-prolog/pack/libssh/build
% make[1]: Entering directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% make[2]: Entering directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% Scanning dependencies of target sshd4pl
% make[2]: Leaving directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% make[2]: Entering directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% [ 50%] Building C object CMakeFiles/sshd4pl.dir/c/sshd4pl.c.o
% [100%] Linking C shared module sshd4pl.so
% make[2]: Leaving directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% [100%] Built target sshd4pl
% make[1]: Leaving directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% make[1]: Entering directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% make[2]: Entering directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% make[2]: Leaving directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% [100%] Built target sshd4pl
% make[1]: Leaving directory '/home/jan/.local/share/swi-prolog/pack/libssh/build'
% Install the project...
% -- Install configuration: ""
% -- Installing: /home/jan/.local/share/swi-prolog/pack/libssh/lib/x86_64-linux/sshd4pl.so
true.

Preparing your ssh setup

We need to configure your ssh setup to be able to login to our server. SSH manages this setup in the directory ~/.ssh. This directory must contain a private key and a matching public key in the file authorized_keys. If you use ssh regularly you probably have a key pair and you only have to ensure the public key is also in authorized_keys. You can do this with e.g.,

cat .ssh/id_rsa.pub >> .ssh/authorized_keys

If you never used ssh you must create the setup from scratch, for example using the commands below. Typically it is wise to set a passphrase on the key.

cd ~
mkdir .ssh
chmod 700 .ssh
ssh-keygen
<default answers, you may add a passphrase or not>
cat .ssh/id_rsa.pub >> .ssh/authorized_keys

Integrating SSHD into your Prolog server

The server is started with ssh_server/1 after loading library(ssh_server). By default it listens to port 2020 on the localhost (127.0.0.1) interface and only allows for SSH public key based login. Options may be used to change this. Thus, normally the following suffices:

:- use_module(library(ssh_server)).
:- initialization(ssh_server([]), program).

A more elaborate example can be found with SWISH in config-available/sshd.pl. The code is bwlow. The listen/2 method is used to start the SSH server after the HTTP daemon software as prepared all the steps to become a service process, notably it may have fork()ed to become a background process. In this example we allow login from any place that can access port 3250 on this machine and we provide an explicit authorized_keys file holding all the public keys of sysadmins and developers that are allowed to login to this service.

:- use_module(library(ssh_server)).
:- use_module(library(broadcast)).

:- listen(http(pre_server_start),
          start_sshd).

start_sshd :-
    ssh_server([ port(3250),
                 bind_address(*),
                 authorized_keys_file('etc/ssh/authorized_keys')
               ]).

Login and logout

After all is setup, simply login using the command below. Adjust the port and host as required.

ssh -p 2020 localhost

Here is a session with a login to swish. It shows the login to non-localhost using a passphrase. Then shows the running threads and a problem: it appears possible that the thread that garbage collects idle HTTP sessions can die, probably as the service temporarily ran out of resources. Near the bottom we examine the status of a thread by asking a backtrace for it.

To log out, use the end-of-file character (normally Control-D) or type

?- end_of_file.

Do not use halt/0!

2_> ssh -p 3250 swish
Enter passphrase for key '/home/janw/.ssh/id_rsa':
X11 forwarding request failed on channel 0
Welcome to SWI-Prolog (threaded, 64 bits, version 8.3.17-6-g8b7f725d4)
SWI-Prolog comes with ABSOLUTELY NO WARRANTY. This is free software.
Please run ?- license. for legal details.

For online help and background, visit https://www.swi-prolog.org
For built-in help, use ?- help(Topic). or ?- apropos(Word).

?- threads.
%                       Thread Status       Time    Stack use    allocated
% ------------------------------------------------------------------------
%                         main running   192.053       76,776      186,176
%                           gc running 29040.846        1,200       87,872
%                  'sshd@3250' running     0.002          872      120,640
%                 'httpd@80_1' running 37484.365       10,352       87,872
%                 'httpd@80_2' running 38823.186       10,352       87,872
%                 'httpd@80_3' running 37921.349       10,352       87,872
%                 'httpd@80_4' running 38052.851       10,352       87,872
%                    'http@80' running   939.398       13,080      120,640
%                  swish_stats running  1646.630      439,104    1,333,056
%           '__http_scheduler' running   214.600       19,472      120,640
%                           12 running     0.000        2,224      120,640
%          '__http_session_gc' exception(error(resource_error(no_memory),context(system:thread_create/3,Resource temporarily unavailable)))
%                   swish_chat running   483.052      429,576      677,696
%      '__thread_pool_manager' running   290.363    1,803,656    2,676,544
%                           29 running     0.027       50,032      120,640
%                           48 running     0.001       11,024      120,640
%  <thread>(53,0x5650b53f46e0) exception(abort_query)
%                           72 running     0.001        3,568      120,640
%                           96 running     0.004       12,008      120,640
% <thread>(113,0x5650b53f4af0) exception(abort_query)
%                          137 running     0.000        2,224      120,640
%                          170 running     0.009       30,360      120,640
%                          190 running     0.001       12,008      120,640
%                          203 running     0.000        8,352      120,640
%                          234 running     0.005        7,144      120,640
%                          247 running     0.001        2,896      120,640
%                          253 running     0.008       56,896      120,640
%             'httpd@80_24424' running     1.762       10,576       87,872
%                          279 running     0.001        3,568      120,640
%                          289 running     0.634   79,887,088   88,088,544
%             'httpd@80_24489' running     0.953       10,576       87,872
%             'httpd@80_24503' running     1.008       10,576       87,872
%                          343 running     0.014        3,016       87,872
%                          349 running     0.017        3,088       87,872
%                          359 running     0.008       30,576      120,640
%              'swish@ssh/389' running     0.002       14,728       87,872
%                          406 running     0.009       11,024      120,640
%                          421 running     0.010       26,528      120,640
%             'httpd@80_24466' running    18.470       10,584       87,872
%             'httpd@80_24488' running     0.834       10,576       87,872
%                          461 running     0.001        3,568      120,640
%                          686 running     0.013       32,952      120,640
%             'httpd@80_24474' running     0.197       10,576       87,872
%                          750 running     0.000        2,224      120,640
% <thread>(752,0x5650b53f6cb0) exception(abort_query)
%             'httpd@80_24479' running     0.106       10,576       87,872
%                          795 running     0.018       69,672      120,640
%             'httpd@80_24469' running     3.198       10,576       87,872
%                          846 running     0.007       27,552      120,640
%                          855 running     0.000        2,224      120,640
%                          860 running     0.005       30,520      120,640
%                          874 running     0.000        2,224      120,640
true.

?- tbacktrace('httpd@80_24424').
   [6] thread_get_message('httpd@80',_24210,[timeout(10)])
   [5] thread_httpd:get_work('httpd@80',_24268,10) at /home/swish/lib/swipl/library/http/thread_httpd.pl:667
   [4] <meta call>
   [3] thread_idle(thread_httpd:get_work('httpd@80',_24332,10),long) <foreign>
   [2] thread_httpd:thread_repeat_wait(thread_httpd:get_work('httpd@80',_24392,10)) at /home/swish/lib/swipl/library/http/thread_httpd.pl:985
   [1] thread_httpd:http_worker([max_idle_time(10),...|...]) at /home/swish/lib/swipl/library/http/thread_httpd.pl:635
   [0] <meta call>
true.

?- ^D
Connection to swish.lxc closed.

Future

Some things on the wish list

Open a break/0 environment in an arbitrary thread
Allow tracing some arbitrary thread, optionally using X11 forwarding to use the graphical debugger.
Support extensions such as sftp to update sources.

Take home

The libssh package provides a secure mechanism to debug and maintain Prolog programs that are not running interactively. This blog post describes how to install and configure the library to get access to a service process. You can also use these features to get access to a Prolog process that is embedded into some C++/Java/Python/... application.