Excessive stunnel logging

Excessive stunnel logging 2 years 11 months ago #2473

Nathan Scannell
Topic Author
Offline
Posts: 38

Hi Salvatore,

I have an issue here with my latest deployment that I'm hoping you can shed some light on.
XCP-ng 8.2, HA Installer version 2.19

The stunnel logging is filling up the logs partition on the slave with the /var/log/secure log file reaching 1.3GB in size.

It appears that the DRBD is causing this.
Is the log level set higher than usual using 2.19 installer?

See attached... Every couple of seconds, the attached log cycle repeats.
192.168.8.231 is the Master
192.168.8.241 is slave.

Cheers,
Nathan

Attachments:

Please Log in or Create an account to join the conversation.

Excessive stunnel logging 2 years 10 months ago #2474

Salvatore Costantino Offline Posts: 722	Hi Nathan, I've experienced the same since migrating some hosts to 8.2. I added the below to crontab of the affected host to suppress the warnings and keep the log partition from filling up. I am not sure what has changed in 8.2 to trigger the increased logging. This will clear rotated logs daily. 0 1 * * * rm -f /var/log/secure.*.gz
	Please Log in or Create an account to join the conversation. Last edit: by Salvatore Costantino.

Excessive stunnel logging 2 years 10 months ago #2475

Nathan Scannell
Topic Author
Offline
Posts: 38

OK... so it looks like they've changed the way they handle SSL tunnels in 8.2 resulting in very verbose logging. Also, SSL Backwards compatibility mode has been disabled.

Deleting old log files is not a viable option... The secure log file fills all remaining space on the log partition. Right now, both my secure and secure.1 are currently 1.5GB each and the partition is only 4GB. Going to need a permanent solution. May have to install a drive just for logs.

The big problem is that the stunnel configuration is essentially hard coded into xapi and so every time xpi restarts, the configuration file is overwritten with default option of 5 (notice)

Currently looking at ways to set level permanently.

Differences below...

7.6
Configruation file here: /etc/xensource/xapi-ssl.conf

; Autogenerated by /opt/xensource/libexec/xapissl
; during xapi start-up.
;
fips = no
pid = /var/run/xapissl.pid
socket = r:TCP_NODELAY=1
socket = a:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
socket = r:SO_KEEPALIVE=1
socket = a:SO_KEEPALIVE=1
compression = zlib


[xapi]
accept = :::443
connect = 80
cert = /etc/xensource/xapi-ssl.pem
ciphers = !SSLv2:RSA+AES128-SHA256:RSA+AES256-SHA:RSA+AES128-SHA:RSA+RC4-SHA:RSA+RC4-MD5:RSA+DES-CBC3-SHA
TIMEOUTclose = 0
options = NO_SSLv2
sslVersion = all

8.2
Configuration file changed to: /etc/stunnel/xapi.conf

; autogenerated by xapi
fips = no
pid = /var/run/xapissl.pid
socket = r:TCP_NODELAY=1
socket = a:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
socket = r:SO_KEEPALIVE=1
socket = a:SO_KEEPALIVE=1
; no idle timeout
debug = authpriv.5

[xapi]
accept = :::443
connect = 80
cert = /etc/xensource/xapi-ssl.pem
ciphers = ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-GCM-SHA384:AES256-SHA256:AES128-SHA256
curve = secp384r1
TIMEOUTclose = 1
options = CIPHER_SERVER_PREFERENCE
sslVersion = TLSv1.2

Please Log in or Create an account to join the conversation.

Last edit: by Nathan Scannell.

Excessive stunnel logging 2 years 10 months ago #2476

Nathan Scannell
Topic Author
Offline
Posts: 38

For immediate mitigation, scheduling the command logrotate-xenserver twice a day should make sure that live log files become compressed before the logs partition becomes full.

All logs will rotate so if longer log persistence is required then the secure log will need to be targeted separately.

/var/log/secure could be separated out from /etc/logrotate.d/syslog and customised with a configuration file of its own but simply running logrotate will not work because compression is done by logrotate-xenserver which affects all logs.

/etc/logrotate.d/syslog

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Please Log in or Create an account to join the conversation.

Last edit: by Nathan Scannell.

Excessive stunnel logging 2 years 8 months ago #2505

Michael Joyner Offline Posts: 5	Look in /etc/cron.d and see if you have a logrotate.cron.rpmsave file without a matching logrotate.cron file: See: github.com/xcp-ng/xcp/issues/510#issuecomment-887581333 The following user(s) said Thank You: Robert Schuh, Nathan Scannell
	Please Log in or Create an account to join the conversation.

Excessive stunnel logging 2 years 8 months ago #2506

Robert Schuh Offline Posts: 4	Thanks Michael, your solution solved the problem. We had multiple XCP-ng 8.1 hosts updated to 8.2 (yum-style upgrade) and each of it was affected by the excessive logging problem afterwards. The file /etc/cron.d/logrotate.cron was missing, so i executed: mv /etc/cron.d/logrotate.cron.rpmsave /etc/cron.d/logrotate.cron and logrotate immediately started compressing the logs.
	Please Log in or Create an account to join the conversation.

TOPIC:

Excessive stunnel logging 2 years 11 months ago #2473

Excessive stunnel logging 2 years 10 months ago #2474

Excessive stunnel logging 2 years 10 months ago #2475

Excessive stunnel logging 2 years 10 months ago #2476

Excessive stunnel logging 2 years 8 months ago #2505

Excessive stunnel logging 2 years 8 months ago #2506

Company

Links

Products