Forum
Welcome, Guest
Username: Password: Remember me
This is the optional category header for the Suggestion Box.
  • Page:
  • 1
  • 2

TOPIC:

Excessive stunnel logging 5 months 1 week ago #2473

  • Konos Koncepts
  • Konos Koncepts's Avatar Topic Author
  • Offline
  • Posts: 41
Hi Salvatore,

I have an issue here with my latest deployment that I'm hoping you can shed some light on.
XCP-ng 8.2, HA Installer version 2.19

The stunnel logging is filling up the logs partition on the slave with the /var/log/secure log file reaching 1.3GB in size.

It appears that the DRBD is causing this.
Is the log level set higher than usual using 2.19 installer?

See attached... Every couple of seconds, the attached log cycle repeats.
192.168.8.231 is the Master
192.168.8.241 is slave.

Cheers,
Nathan
Attachments:

Please Log in or Create an account to join the conversation.

Excessive stunnel logging 5 months 1 week ago #2474

  • Salvatore Costantino
  • Salvatore Costantino's Avatar
  • Offline
  • Posts: 703
Hi Nathan,
I've experienced the same since migrating some hosts to 8.2.

I added the below to crontab of the affected host to suppress the warnings and keep the log partition from filling up. I am not sure what has changed in 8.2 to trigger the increased logging. This will clear rotated logs daily.
0 1 * * * rm -f /var/log/secure.*.gz

Please Log in or Create an account to join the conversation.

Last edit: by Salvatore Costantino.

Excessive stunnel logging 5 months 1 week ago #2475

  • Konos Koncepts
  • Konos Koncepts's Avatar Topic Author
  • Offline
  • Posts: 41
OK... so it looks like they've changed the way they handle SSL tunnels in 8.2 resulting in very verbose logging. Also, SSL Backwards compatibility mode has been disabled.

Deleting old log files is not a viable option... The secure log file fills all remaining space on the log partition. Right now, both my secure and secure.1 are currently 1.5GB each and the partition is only 4GB. Going to need a permanent solution. May have to install a drive just for logs.

The big problem is that the stunnel configuration is essentially hard coded into xapi and so every time xpi restarts, the configuration file is overwritten with default option of 5 (notice)

Currently looking at ways to set level permanently.

Differences below...



7.6
Configruation file here: /etc/xensource/xapi-ssl.conf
; Autogenerated by /opt/xensource/libexec/xapissl
; during xapi start-up.
;
fips = no
pid = /var/run/xapissl.pid
socket = r:TCP_NODELAY=1
socket = a:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
socket = r:SO_KEEPALIVE=1
socket = a:SO_KEEPALIVE=1
compression = zlib


[xapi]
accept = :::443
connect = 80
cert = /etc/xensource/xapi-ssl.pem
ciphers = !SSLv2:RSA+AES128-SHA256:RSA+AES256-SHA:RSA+AES128-SHA:RSA+RC4-SHA:RSA+RC4-MD5:RSA+DES-CBC3-SHA
TIMEOUTclose = 0
options = NO_SSLv2
sslVersion = all



8.2
Configuration file changed to: /etc/stunnel/xapi.conf
; autogenerated by xapi
fips = no
pid = /var/run/xapissl.pid
socket = r:TCP_NODELAY=1
socket = a:TCP_NODELAY=1
socket = l:TCP_NODELAY=1
socket = r:SO_KEEPALIVE=1
socket = a:SO_KEEPALIVE=1
; no idle timeout
debug = authpriv.5

[xapi]
accept = :::443
connect = 80
cert = /etc/xensource/xapi-ssl.pem
ciphers = ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-GCM-SHA384:AES256-SHA256:AES128-SHA256
curve = secp384r1
TIMEOUTclose = 1
options = CIPHER_SERVER_PREFERENCE
sslVersion = TLSv1.2

Please Log in or Create an account to join the conversation.

Last edit: by Konos Koncepts.

Excessive stunnel logging 5 months 1 week ago #2476

  • Konos Koncepts
  • Konos Koncepts's Avatar Topic Author
  • Offline
  • Posts: 41
For immediate mitigation, scheduling the command logrotate-xenserver twice a day should make sure that live log files become compressed before the logs partition becomes full.

All logs will rotate so if longer log persistence is required then the secure log will need to be targeted separately.

/var/log/secure could be separated out from /etc/logrotate.d/syslog and customised with a configuration file of its own but simply running logrotate will not work because compression is done by logrotate-xenserver which affects all logs.

/etc/logrotate.d/syslog
/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Please Log in or Create an account to join the conversation.

Last edit: by Konos Koncepts.

Excessive stunnel logging 2 months 4 weeks ago #2505

Look in /etc/cron.d and see if you have a logrotate.cron.rpmsave file without a matching logrotate.cron file:

See: github.com/xcp-ng/xcp/issues/510#issuecomment-887581333
The following user(s) said Thank You: Robert Schuh, Konos Koncepts

Please Log in or Create an account to join the conversation.

Excessive stunnel logging 2 months 4 weeks ago #2506

Thanks Michael,

your solution solved the problem.
We had multiple XCP-ng 8.1 hosts updated to 8.2 (yum-style upgrade) and each of it was affected by the excessive logging problem afterwards.

The file /etc/cron.d/logrotate.cron was missing, so i executed:
mv /etc/cron.d/logrotate.cron.rpmsave /etc/cron.d/logrotate.cron
and logrotate immediately started compressing the logs.

Please Log in or Create an account to join the conversation.

  • Page:
  • 1
  • 2