journalctl very slow

[Ruediger Meier]

Hi,

I'am trying to watch the logs but it's a bit too slow.

journalctl

is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?
journalctl also consumes 100% CPU the whole time so the machine is
useless currently.


cu,
Rudi

[Cristian Rodríguez]

Hi,

Post by Ruediger Meier
I'am trying to watch the logs but it's a bit too slow.

Well Known issue, no solution for now. and if I were you I won't be
holding my breath for one to be provided anytime soon. (last discussed
in early Oct @systemd-devel thread title Fwd: Journalctl performance)

This happens almost exclusively when the journal is stored in rotating
media.

If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

[Yamaban]

Hi,

Post by Ruediger Meier
I'am trying to watch the logs but it's a bit too slow.

Well Known issue, no solution for now. and if I were you I won't be holding
my breath for one to be provided anytime soon. (last discussed in early Oct
@systemd-devel thread title Fwd: Journalctl performance)
This happens almost exclusively when the journal is stored in rotating media.
If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

Niggle: isn't that /etc/systemd/journald.conf ?
Also, see: man 5 journald.conf

But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.

Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

- Yamaban.

[Archie Cobbs]

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

Maybe this should be the openSUSE default.

I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

-Archie

[Ruediger Meier]

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in
charge, sounds of '... rotating media are a dying type ...' where
made, when asked on the topic during a Linux conf earlier this
year.
Do not hope for a solution from the systemd / journald side,
install a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

Where was already a big discussion before syslog was removed at a time
when journald was still even more broken than now. As usualy "the
decision has been made" anyway. Sysadmins are not openSUSE targets.

cu,
Rudi

[Cristian Rodríguez]

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

If you install a traditional syslog implementation there will be no
difference.. also people keep repeating "opaqueness and cumbersomeness"
but never care to elaborate exactly what they mean in a detailed form
and compared to what.

That said, embedded developers from Samsung and Intel do care about this
slowness and are working on it (much better approach than whining and
complaining I must add)

[Ken Schneider - openSUSE]

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

If you install a traditional syslog implementation there will be no
difference.. also people keep repeating "opaqueness and cumbersomeness"
but never care to elaborate exactly what they mean in a detailed form
and compared to what.
That said, embedded developers from Samsung and Intel do care about this
slowness and are working on it (much better approach than whining and
complaining I must add)

It has become the means of complaining since the elitist devs won't
bother to look at bug reports without just marking them as "Won't Fix"
or "to be fixed in a future version (AKA no one knows when)". People
have been complaining about logging for a long time and now we're told
"too bad, we don't care about sysadmins".

[Andrey Borzenkov]

В Fri, 01 Nov 2013 14:03:09 -0300

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

If you install a traditional syslog implementation there will be no
difference..

Then it should be default setting in openSUSE until problem with
journald slowness is fixed, should not it?

Post by Cristian Rodríguez
also people keep repeating "opaqueness and cumbersomeness"
but never care to elaborate exactly what they mean in a detailed form
and compared to what.
That said, embedded developers from Samsung and Intel do care about this
slowness and are working on it (much better approach than whining and
complaining I must add)

The question was "why it became default when it had obvious problems";
do not pretend you do not see the difference. It is not about
development of code, it is about selecting distribution default
behavior.

[Cristian Rodríguez]

Post by Andrey Borzenkov
The question was "why it became default when it had obvious problems";
do not pretend you do not see the difference. It is not about
development of code, it is about selecting distribution default
behavior.

Last time I did a clean installation, package systemd-logger was not
installed by default (neither is listed to be installed in current
opensuse-patterns) therefore no persistent journal, nothing gets written
to /var/log/journal..(directory absent) and I had to remove rsyslog in
order to only use the journal. (they conflict with each other)


zypper in rsyslog
Loading repository data...
Reading installed packages...
Resolving package dependencies...

Problem: systemd-logger-208-6.1.x86_64 conflicts with
namespace:otherproviders(syslog) provided by rsyslog-7.4.4-2.1.2.x86_64
Solution 1: deinstallation of systemd-logger-208-6.1.x86_64
Solution 2: do not install rsyslog-7.4.4-2.1.2.x86_64

(this is technically wrong but well..)

If this has changed somehow.. I do not know..

[Greg Freemyer]

Post by Andrey Borzenkov
В Fri, 01 Nov 2013 14:03:09 -0300

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in

charge,

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
sounds of '... rotating media are a dying type ...' where made,

when

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side,

install

Post by Cristian Rodríguez

Post by Archie Cobbs

Post by Yamaban
a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

If you install a traditional syslog implementation there will be no
difference..

Then it should be default setting in openSUSE until problem with
journald slowness is fixed, should not it?

Post by Cristian Rodríguez
also people keep repeating "opaqueness and

cumbersomeness"

Post by Cristian Rodríguez
but never care to elaborate exactly what they mean in a detailed form
and compared to what.
That said, embedded developers from Samsung and Intel do care about

this

Post by Cristian Rodríguez
slowness and are working on it (much better approach than whining and
complaining I must add)

The question was "why it became default when it had obvious problems";
do not pretend you do not see the difference. It is not about
development of code, it is about selecting distribution default
behavior.

Since it is relatively small, is there a way to page it into ram before starting to parse it.

Ie. Add a patch to do a linear read of the entire fire before starting the random I/O.

For systems with some spare ram that should drastically accelerate parsing the file and for systems tight in ram it would not hurt much.

Greg

[Jan Engelhardt]

Post by Greg Freemyer
Since it is relatively small, is there a way to page it into ram before
starting to parse it. Ie. Add a patch to do a linear read of the entire fire
before starting the random I/O. For systems with some spare ram that should
drastically accelerate parsing the file and for systems tight in ram it would
not hurt much.

How small is small? And if it's "big enough", it will be evicted from
the cache sooner than you think. So no, that does not help.

[Cristian Rodríguez]

Post by Andrey Borzenkov
The question was "why it became default when it had obvious problems";
do not pretend you do not see the difference. It is not about
development of code, it is about selecting distribution default
behavior.

Ok, I just picked openSUSE 13.1 RC2 KDE Live and installed it on a
virtual machine

- rsyslog is installed by default.
- there is no systemd-logger installed, therefore no /var/log/journal,
therefore no persistent journal.

This is exactly like there was before, the default has not been changed,
nothing needs to be documented in the release notes. etc.

I do not know where people got this idea in the first place.

[Guido Berhoerster]

Post by Cristian Rodríguez

Post by Andrey Borzenkov
The question was "why it became default when it had obvious problems";
do not pretend you do not see the difference. It is not about
development of code, it is about selecting distribution default
behavior.

Ok, I just picked openSUSE 13.1 RC2 KDE Live and installed it on a
virtual machine
- rsyslog is installed by default.
- there is no systemd-logger installed, therefore no
/var/log/journal, therefore no persistent journal.
This is exactly like there was before, the default has not been
changed, nothing needs to be documented in the release notes. etc.
I do not know where people got this idea in the first place.

https://github.com/openSUSE/patterns/pull/58

[Cristian Rodríguez]

Post by Guido Berhoerster
https://github.com/openSUSE/patterns/pull/58

Yeah, not merged yet.. it says "after 13.1"

[Ruediger Meier]

Post by Cristian Rodríguez

Hi,

Post by Ruediger Meier
I'am trying to watch the logs but it's a bit too slow.

Well Known issue, no solution for now. and if I were you I won't be
holding my breath for one to be provided anytime soon. (last
performance)

How could this journald logging daemon got released and (used in
openSUSE per default) if it's not possible to view the logs at all?

Post by Cristian Rodríguez
This happens almost exclusively when the journal is stored in
rotating media.
If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

I have rsyslog installed but I've been told to watch the journal to
debug initrd problems. But this is actually not possible.

BTW my production machines (still all 11.4) have average uptimes between
200 and 400 days. So I expect that the "current boot logs" would be as
big as this one on my test machine.
The journal disk-usage is currently 1.3G, which is far to much. The same
as compressed text files would be probably about 20M (factor 60!).

It really does not make sense that syslog was removed per default and
replaced by such a monster which is using 60 times more space and we
are not even able to view that log unless we want to wait a whole week
while the machine is unusable on 100% load.

cu,
Rudi

[Cristian Rodríguez]

Post by Ruediger Meier
How could this journald logging daemon got released and (used in
openSUSE per default) if it's not possible to view the logs at all?

The logs can be viewed, it is just slow with non-SSD drives.. it has to
be used by default because it is mandatory ! I have explained this I do
not know how many times.

Post by Ruediger Meier
I have rsyslog installed but I've been told to watch the journal to
debug initrd problems. But this is actually not possible.

ALL messages that the journal captures are forwarded to the syslog
implementation if installed.

Post by Ruediger Meier
It really does not make sense that syslog was removed per default

according to the patterns file, rsyslog is still a suggested package, it
is not required however.

[Yamaban]

Post by Ruediger Meier
How could this journald logging daemon got released and (used in
openSUSE per default) if it's not possible to view the logs at all?

The logs can be viewed, it is just slow with non-SSD drives.. it has to be
used by default because it is mandatory ! I have explained this I do not know
how many times.

Could we put a short info on that (journald slow with non-SSD drives) and
the "should have" package rsyslog or similar for server use into the
Release Notes, for the search term "logging" ?

Post by Ruediger Meier
I have rsyslog installed but I've been told to watch the journal to
debug initrd problems. But this is actually not possible.

ALL messages that the journal captures are forwarded to the syslog
implementation if installed.

Also well worth mentioning, see above.

- Yamaban

[Jan Engelhardt]

Post by Yamaban

Post by Ruediger Meier
How could this journald logging daemon got released and (used in
openSUSE per default) if it's not possible to view the logs at all?

The logs can be viewed, it is just slow with non-SSD drives.. it has to be
used by default because it is mandatory ! I have explained this I do not know
how many times.

Could we put a short info on that (journald slow with non-SSD drives) and
the "should have" package rsyslog or similar for server use into the
Release Notes, for the search term "logging" ?

Even with rsyslog, `systemctl status` searches the journal which can
take noticable time.

[Rajko]

On Fri, 1 Nov 2013 17:57:28 +0100 (CET)

Post by Yamaban
Could we put a short info on that (journald slow with non-SSD drives)
and the "should have" package rsyslog or similar for server use into
the Release Notes, for the search term "logging" ?

Release Notes is better handled via bugzilla (for now).

Product: openSUSE 13.1
Component: Release Notes

That way documentation guys will get message.
For cross reference, you can list bug report URL here, and thread
reporting proposal for change in the bug report.

[Per Jessen]

Post by Cristian Rodríguez

Post by Ruediger Meier
How could this journald logging daemon got released and (used in
openSUSE per default) if it's not possible to view the logs at all?

The logs can be viewed, it is just slow with non-SSD drives.. it has
to be used by default because it is mandatory ! I have explained this
I do not know how many times.

Post by Ruediger Meier
I have rsyslog installed but I've been told to watch the journal to
debug initrd problems. But this is actually not possible.

ALL messages that the journal captures are forwarded to the syslog
implementation if installed.

Post by Ruediger Meier
It really does not make sense that syslog was removed per default

according to the patterns file, rsyslog is still a suggested package,
it is not required however.

https://bugzilla.novell.com/show_bug.cgi?id=843657

[Cristian Rodríguez]

Post by Ruediger Meier
The journal disk-usage is currently 1.3G, which is far to much.

Far too much..in what kind of system .. do you have harddisks from the
90's ? in 2011, the average *consumer* hard drive was 590 GB..
increasing size at a rate of 39% per year.


The same

Post by Ruediger Meier
as compressed text files would be probably about 20M (factor 60!).

The journal is already compressed (XZ is used.. ) it is of course much
bigger than a text file because it has metadata attached.

[Ruediger Meier]

Post by Cristian Rodríguez

Post by Ruediger Meier
The journal disk-usage is currently 1.3G, which is far to much.

Far too much..in what kind of system .. do you have harddisks from
the 90's ? in 2011, the average *consumer* hard drive was 590 GB..
increasing size at a rate of 39% per year.

How ignorant and blind are you ...

Journalctl showing the logs with 2K/s - that's 60's style

In 2013 systemd-journal (the default in most distros :) needs a week to
show such a 1.3G journal ... it can't even handle 90's file sizes but
you complain that users don't always want to keep their / small.


vm servers are hosting dozens or hundreds of journald "powered" systems.
They all need much more space / network bandwidth / traffic / time /
energy to backup.

This is one of our typical productive openSUSE 11.4 installations:
Filesystem Size Used Avail Use% Mounted on
rootfs 3.0G 1.1G 1.8G 38% /
devtmpfs 243M 88K 243M 1% /dev
tmpfs 246M 4.0K 246M 1% /dev/shm
/dev/vda1 3.0G 1.1G 1.8G 38% /

Upgrading to journald per default means either only keeping logs for a
few last days or double hd space.

We have rpmlint complaining about a few KB documentation in non-doc
packages (and much more similar warnings). Not even significant when
journald is in use. But not allowed to complain about a journal wasting
giga bytes?

Even you always try to convince us that that tmpfs (maybe < 4-8GB on
average systems) are enough /tmp for everybody. How does this
match? /var/log has to be 201x style but /tmp must be 90' style?

We have still many NFS client desktops with <80GB HD where / is ~20GB
(90% full) and thre rest is local /tmp storage for the developers and
number crunchers. Each wasted GB would be painful for our users and
then for our NFS server.

Post by Cristian Rodríguez
The same

Post by Ruediger Meier
as compressed text files would be probably about 20M (factor 60!).

The journal is already compressed (XZ is used.. ) it is of course
much bigger than a text file because it has metadata attached.

Haha.


cu,
Rudi

[Cristian Rodríguez]

Post by Ruediger Meier

Post by Cristian Rodríguez

Post by Ruediger Meier
The journal disk-usage is currently 1.3G, which is far to much.

Far too much..in what kind of system .. do you have harddisks from
the 90's ? in 2011, the average *consumer* hard drive was 590 GB..
increasing size at a rate of 39% per year.

How ignorant and blind are you ...

So you disagree with me..:-)

Post by Ruediger Meier
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to
show such a 1.3G journal ... it can't even handle 90's file sizes but
you complain that users don't always want to keep their / small.

if 1.3G is too big for your usecase, you can configure the maximum size
that the journal can use on disk, or configure it to only retain the
logs of the current boot at a particular limit (embedded developers do
this all the time as they have limited ram, slower media) or set the
Journal storage to none, which is not recommended and then you must have
a syslog implementation installed .you will loose pretty much all
systemd functionality that depends on logging.

Post by Ruediger Meier
We have rpmlint complaining about a few KB documentation in non-doc
packages (and much more similar warnings). Not even significant when
journald is in use. But not allowed to complain about a journal wasting
giga bytes?

Big difference, the size of the journal does not affect the size of the
installation media, that's what rpmlint cares about.

Post by Ruediger Meier
Even you always try to convince us that that tmpfs (maybe < 4-8GB on
average systems) are enough /tmp for everybody.

Not for everybody, but for the average use, it is not the default and
has nothing to do with journal either.

[Claudio Freire]

Post by Ruediger Meier

Post by Cristian Rodríguez

Post by Ruediger Meier
The journal disk-usage is currently 1.3G, which is far to much.

Far too much..in what kind of system .. do you have harddisks from
the 90's ? in 2011, the average *consumer* hard drive was 590 GB..
increasing size at a rate of 39% per year.

How ignorant and blind are you ...
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to
show such a 1.3G journal ... it can't even handle 90's file sizes but
you complain that users don't always want to keep their / small.

This is because journal is using memory-mapped access.

Memory-mapped access is terribly bad at predicting I/O patterns. This
may be a kernel issue more than a journal issue, but the journal can
help the kernel predict better.

I suggest monitorion I/O with iostat to see what the issue is. There
should be a lot of read-request-merges since watching a journal log
should be sequential access. If read-request-merges are low and avg
request sizes too, then it's probably fragmentation.

I'll be happy to provide a few patches for you to try trying to
address that, if you're willing to rebuild and test.

[Claudio Freire]

Post by Claudio Freire
I suggest monitorion I/O with iostat to see what the issue is. There
should be a lot of read-request-merges since watching a journal log
should be sequential access. If read-request-merges are low and avg
request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to
address that, if you're willing to rebuild and test.

Or better yet. Send me the journal file somehow?

That way I can test myself.

[Ruediger Meier]

On Sun, Nov 3, 2013 at 3:42 PM, Claudio Freire

Post by Claudio Freire
I suggest monitorion I/O with iostat to see what the issue is.
There should be a lot of read-request-merges since watching a
journal log should be sequential access. If read-request-merges are
low and avg request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to
address that, if you're willing to rebuild and test.

Or better yet. Send me the journal file somehow?
That way I can test myself.

Thanks a lot for your efforts but unfortunately I had removed that
useless journal stuff already.
Actually I also don't really believe that you could accelerate it by
factor 1000 to make it at least a bit more useful. Maybe it's this
hopeless looking situation why upstream does not "want" to fix that,
instead telling us that our needs or hardware is wrong.

cu,
Rudi

[Claudio Freire]

Post by Ruediger Meier

On Sun, Nov 3, 2013 at 3:42 PM, Claudio Freire

Post by Claudio Freire
I suggest monitorion I/O with iostat to see what the issue is.
There should be a lot of read-request-merges since watching a
journal log should be sequential access. If read-request-merges are
low and avg request sizes too, then it's probably fragmentation.
I'll be happy to provide a few patches for you to try trying to
address that, if you're willing to rebuild and test.

Or better yet. Send me the journal file somehow?
That way I can test myself.

Thanks a lot for your efforts but unfortunately I had removed that
useless journal stuff already.
Actually I also don't really believe that you could accelerate it by
factor 1000 to make it at least a bit more useful. Maybe it's this
hopeless looking situation why upstream does not "want" to fix that,
instead telling us that our needs or hardware is wrong.

Well, without knowing what the bottleneck is, it's hard to tell. My
money was either I/O slowness due to fragmentation (easy to fix with a
posix_fallocate call) or repeated decompression overhead (fixed not so
easily but still easy-ish with a block cache). With a patch that
addresses the bottleneck, I bet upstream would pay attention.

[Carlos E. R.]

This happens almost exclusively when the journal is stored in rotating media.
If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

Eleanor4:~ # less /etc/systemd/journal.conf
/etc/systemd/journal.conf: No such file or directory
Eleanor4:~ #
Eleanor4:~ # cat /etc/os-release
NAME=openSUSE
VERSION="13.1 (Bottle)"
VERSION_ID="13.1"
PRETTY_NAME="openSUSE 13.1 (Bottle) (x86_64)"
ID=opensuse
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:opensuse:13.1"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://opensuse.org/"
ID_LIKE="suse"
Eleanor4:~ #
Eleanor4:~ # rpm -q rsyslog
rsyslog-7.4.4-2.1.2.x86_64
Eleanor4:~ #
Eleanor4:~ # locate journal.conf
Eleanor4:~ # locate -i journal.conf
Eleanor4:~ #

Default install...

- --
Cheers,
Carlos E. R.
(from 12.3 x86_64 "Dartmouth" at Telcontar)

[Carlos E. R.]

Post by Carlos E. R.

Post by Cristian Rodríguez
If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

Eleanor4:~ # less /etc/systemd/journal.conf
/etc/systemd/journal.conf: No such file or directory

It is /etc/systemd/journald.conf


- --
Cheers,
Carlos E. R.
(from 12.3 x86_64 "Dartmouth" at Telcontar)

[Cristian Rodríguez]

Post by Carlos E. R.

Post by Carlos E. R.

Post by Cristian Rodríguez
If this bothers you, just set Storage=volatile in in
/etc/systemd/journal.conf and if you need to retain the
logs for more than the current boot, install rsyslog.

Eleanor4:~ # less /etc/systemd/journal.conf
/etc/systemd/journal.conf: No such file or directory

It is /etc/systemd/journald.conf

Yes, I missed a "d" in the mail, I guess people can figure that out
themselves no ? ;-P

[Stefan Seyfried]

Post by Ruediger Meier
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?

Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance of the
"database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

[Yamaban]

Post by Stefan Seyfried

Post by Ruediger Meier
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance of the
"database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Shall we hope you've used the [sarcasm] tag for your reply?

TBH, I've been looking at the journald code wrt the on disk ops,
and, my impression was: Somebody took code for a strict In-RAM
data structure and used that for the On-Disk ops.

IMHO a no-go for any 'real' core OS developer, but, well, look
at who's in charge of that project upstream....

For me, the journald storage code is a thing to be
replaced, at whole.

Linus would never allow that level of code quality
anywhere in the Kernel tree.

No other syslog implementation is done that bad.

And, no, I'm no C / C++ programmer, I'll keep my hand off that.

- Yamaban.

[Cristian Rodríguez]

Post by Yamaban
IMHO a no-go for any 'real' core OS developer, but, well, look
at who's in charge of that project upstream....

And that's your argument, an ad-homimem attack.. all nicely bundled with
a "not true scotsman fallacy" .,. is that the best you can do ?

Post by Yamaban
Linus would never allow that level of code quality
anywhere in the Kernel tree.

Linus has no authority in userspace, he is just another guy with an
opinion..

Post by Yamaban
No other syslog implementation is done that bad.

The journal is not a syslog implementation. comparing pears with apples.

[Ken Schneider - openSUSE]

Post by Cristian Rodríguez

Post by Yamaban
IMHO a no-go for any 'real' core OS developer, but, well, look
at who's in charge of that project upstream....

And that's your argument, an ad-homimem attack.. all nicely bundled with
a "not true scotsman fallacy" .,. is that the best you can do ?

Post by Yamaban
Linus would never allow that level of code quality
anywhere in the Kernel tree.

Linus has no authority in userspace, he is just another guy with an
opinion..

Post by Yamaban
No other syslog implementation is done that bad.

The journal is not a syslog implementation. comparing pears with apples.

Touchy touchy are we. When a piece of software is crap it is crap. No
need for a PhD here to see the results.

[Claudio Freire]

On Fri, Nov 1, 2013 at 8:17 PM, Cristian Rodríguez

Post by Yamaban
IMHO a no-go for any 'real' core OS developer, but, well, look
at who's in charge of that project upstream....

And that's your argument, an ad-homimem attack.. all nicely bundled with a
"not true scotsman fallacy" .,. is that the best you can do ?

Don't be negligently dismissive. He's on target. It's a bad data
structure for disk access, it was a bad idea, and it's just a matter
of the devs getting the necessary years of woes to be able to give up
their egos and accept that.

Nevertheless, my previous mail stands. It's possible to improve it
without much effort, by telling the kernel what to expect.



On Fri, Nov 1, 2013 at 3:53 PM, Cristian Rodríguez

Post by Yamaban
Journalctl showing the logs with 2K/s - that's 60's style
In 2013 systemd-journal (the default in most distros :) needs a week to
show such a 1.3G journal ... it can't even handle 90's file sizes but
you complain that users don't always want to keep their / small.

if 1.3G is too big for your usecase, you can configure the maximum size that
the journal can use on disk, or configure it to only retain the logs of the
current boot at a particular limit (embedded developers do this all the time
as they have limited ram, slower media) or set the Journal storage to none,
which is not recommended and then you must have a syslog implementation
installed .you will loose pretty much all systemd functionality that depends
on logging.

I think the amount of metadata should also be configurable, if that's
the main reason for bloat. I suspect it's not the case though.

Also, having the journal, a memory-mapped write-heavy structure, on an
SSD, is a good way to burn it prematurely.

Post by Yamaban

Post by Archie Cobbs

Post by Yamaban
But, yes, clear upstream trouble, that is: ignored by those in charge,
sounds of '... rotating media are a dying type ...' where made, when
asked on the topic during a Linux conf earlier this year.
Do not hope for a solution from the systemd / journald side, install
a full 'traditional' syslog.

Maybe this should be the openSUSE default.
I for one am very bothered by the way systemd has made logging more
opaque and cumbersome. Logs are like bread and water to sysadmins.

If you install a traditional syslog implementation there will be no
difference..

Then it should be default setting in openSUSE until problem with
journald slowness is fixed, should not it?

Yes. I think it is, in fact. Or at least seemed to last time I
installed 13.1 RC1. Will have to double-check.

[Carlos E. R.]

Post by Stefan Seyfried

Post by Ruediger Meier
is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance of the
"database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

[Greg KH]

Post by Carlos E. R.

Post by Stefan Seyfried

Post by Ruediger Meier
is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance of the
"database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

Really? Patches are always gladly accepted upstream for this, and
numerous people have looked at it to try to firstly determine what
exactly the problem is, and no one has been able to figure that out.
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)

So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

greg k-h

[Ruediger Meier]

Post by Greg KH

Post by Carlos E. R.

Post by Stefan Seyfried

Post by Ruediger Meier
is running since 8 hours now and it has printed only 120MB text
yet. That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance
of the "database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

Really? Patches are always gladly accepted upstream for this, and
numerous people have looked at it to try to firstly determine what
exactly the problem is, and no one has been able to figure that out.
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

Yes probably the design is more bad than the actual implementation.

Anyway, it was very easily predictable that we would get problems like
this. All I complain about is that we (the users/admins) get such
young, untested, unstable, broken stuff per default.

There is already a distribution (Fedora) which is testing all this alpha
software. They have never cared about compatibility or the upgrade
scenario - that's ok because you know that. Why openSUSE does not just
wait until at least one distribution exists where all this funny
freedesktop stuff works as it's supposed too work?

cu,
Rudi

[Greg KH]

Post by Ruediger Meier

Post by Greg KH

Post by Carlos E. R.

Post by Stefan Seyfried

Post by Ruediger Meier
is running since 8 hours now and it has printed only 120MB text
yet. That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance
of the "database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

Really? Patches are always gladly accepted upstream for this, and
numerous people have looked at it to try to firstly determine what
exactly the problem is, and no one has been able to figure that out.
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

Yes probably the design is more bad than the actual implementation.
Anyway, it was very easily predictable that we would get problems like
this. All I complain about is that we (the users/admins) get such
young, untested, unstable, broken stuff per default.

Yes, we all write crappy software and have no idea what we are doing and
should listen to everyone who tells us to stop because they are the ones
who know best.

Give me a break.

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this alpha
software. They have never cared about compatibility or the upgrade
scenario - that's ok because you know that. Why openSUSE does not just
wait until at least one distribution exists where all this funny
freedesktop stuff works as it's supposed too work?

It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo). If you really
hate this type of thing, and don't want progress, there's always
Slackware. As it is, openSUSE is already playing catch-up with those
distros, how far behind do you want to see us be?

greg k-h

[Per Jessen]

[snip]

Post by Greg KH

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this
alpha software. They have never cared about compatibility or the
upgrade scenario - that's ok because you know that. Why openSUSE does
not just wait until at least one distribution exists where all this
funny freedesktop stuff works as it's supposed too work?

It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo). If you
really hate this type of thing, and don't want progress, there's
always Slackware. As it is, openSUSE is already playing catch-up with
those distros, how far behind do you want to see us be?

Playing catch-up is fine with me, trailing edge is a much safer place to
be. Anyway, what does our mission statement say?

[Greg KH]

Post by Per Jessen
[snip]

Post by Greg KH

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this
alpha software. They have never cared about compatibility or the
upgrade scenario - that's ok because you know that. Why openSUSE does
not just wait until at least one distribution exists where all this
funny freedesktop stuff works as it's supposed too work?

It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo). If you
really hate this type of thing, and don't want progress, there's
always Slackware. As it is, openSUSE is already playing catch-up with
those distros, how far behind do you want to see us be?

Playing catch-up is fine with me, trailing edge is a much safer place to
be. Anyway, what does our mission statement say?

"Have a lot of fun..."

[Per Jessen]

Post by Greg KH

Post by Per Jessen
[snip]

Post by Greg KH

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this
alpha software. They have never cared about compatibility or the
upgrade scenario - that's ok because you know that. Why openSUSE
does not just wait until at least one distribution exists where
all this funny freedesktop stuff works as it's supposed too work?

It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo). If you
really hate this type of thing, and don't want progress, there's
always Slackware. As it is, openSUSE is already playing catch-up
with those distros, how far behind do you want to see us be?

Playing catch-up is fine with me, trailing edge is a much safer place
to be. Anyway, what does our mission statement say?

"Have a lot of fun..."

Haha, good point! That's not the whole answer though. One of these days
we ought to address that question thoroughly.

[Ruediger Meier]

Post by Greg KH

Post by Ruediger Meier

Post by Greg KH

Post by Carlos E. R.

Post by Stefan Seyfried

Post by Ruediger Meier
is running since 8 hours now and it has printed only 120MB
text yet. That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the
performance of the "database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

Really? Patches are always gladly accepted upstream for this,
and numerous people have looked at it to try to firstly determine
what exactly the problem is, and no one has been able to figure
that out. Including the fact that most of us who have looked at
it, can't even duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial
problem for a small number of systems that people are not
ignoring...

Yes probably the design is more bad than the actual implementation.
Anyway, it was very easily predictable that we would get problems
like this. All I complain about is that we (the users/admins) get
such young, untested, unstable, broken stuff per default.

Yes, we all write crappy software and have no idea what we are doing
and should listen to everyone who tells us to stop because they are
the ones who know best.
Give me a break.

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this
alpha software. They have never cared about compatibility or the
upgrade scenario - that's ok because you know that. Why openSUSE
does not just wait until at least one distribution exists where all
this funny freedesktop stuff works as it's supposed too work?

It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo).

Does Gentoo and Arch really force you to switch to systemd yet?

Post by Greg KH
If you
really hate this type of thing, and don't want progress, there's
always Slackware. As it is, openSUSE is already playing catch-up
with those distros, how far behind do you want to see us be?

I don't say something against shipping "progress" but against changing
well tested and stable defaults far to early.

For example there was absolutely no need that upgrading from to 12.1 or
12.2 replaced sysvinit by systemd. There would have been enough
pro-active testers. No need to bother everybody.

cu,
Rudi

[Greg KH]

Post by Ruediger Meier

Post by Greg KH
It works just fine for me here, on three other distros who have been
shipping it for a long time (Fedora and Arch and Gentoo).

Does Gentoo and Arch really force you to switch to systemd yet?

Arch switched a long time ago, and Gentoo makes you do it if you want to
run GNOME, which is fine with me.

Post by Ruediger Meier

Post by Greg KH
If you
really hate this type of thing, and don't want progress, there's
always Slackware. As it is, openSUSE is already playing catch-up
with those distros, how far behind do you want to see us be?

I don't say something against shipping "progress" but against changing
well tested and stable defaults far to early.

Please define "early" in a way that is sufficient for everyone.

Post by Ruediger Meier
For example there was absolutely no need that upgrading from to 12.1 or
12.2 replaced sysvinit by systemd. There would have been enough
pro-active testers. No need to bother everybody.

How much "bother" was it really? And how would delaying cause that
"bother" to go away except to postpone it?

Anyway, that's not the issue here at all, we aren't going to rehash the
systemd issue anymore.

Again, if you don't like it, there are still distros out there without
systemd, feel free to use one of them if you like. I hear some people
like Ubuntu these days...

Best of luck,

greg k-h

[Cristian Rodríguez]

Post by Ruediger Meier
There is already a distribution (Fedora) which is testing all this alpha
software. They have never cared about compatibility or the upgrade
scenario - that's ok because you know that. Why openSUSE does not just
wait until at least one distribution exists where all this funny
freedesktop stuff works as it's supposed too work?

In that case we will not be able to influence any design decision, any
of the features, etc. essentially we will be the last to attendant to
arrive to the party and will have to plumb our way up in the most
painful manner in existence. In reality, we cannot afford to be late,
due to a number of factors, including resource constrains.

If you expect things to work the way you just told, I am very sorry to
inform you that things will get much, much worst for you in the near
future and you might want to consider a dying OS family like the BSDs.

Previously I would have recommended you Slackware, but recently Pat V
stated that he is not ruling out switching to systemd if necessary,
however he did rule out going to openRC.

[Stefan Seyfried]

Post by Greg KH
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)

Just try to use it on old (slow) rotating rust.
100% reproducible on all my 2.5" IDE HDDs here.

If I were to guess, I'd vote it's just horribly fragmenting the files:
shake -vvv shows thousands of fragments, probably due to lots of holes
in the database files.

Post by Greg KH
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

The developers of core system software should be forced to use 10 year
old hardware to test their creation in daily use ;-)

[Ruediger Meier]

Post by Stefan Seyfried

Post by Greg KH
Including the fact that most of us who have looked at it, can't
even duplicate the issue at all (myself included.)

Just try to use it on old (slow) rotating rust.
100% reproducible on all my 2.5" IDE HDDs here.

BTW my problem does not looked like HD speed issue. journalctl was
constantly on 100% CPU. So I assume that it couldn't handle more input
data in the same time (unless it's even more broken than thought).

Post by Stefan Seyfried
If I were to guess, I'd vote it's just horribly fragmenting the
files: shake -vvv shows thousands of fragments, probably due to lots
of holes in the database files.

Post by Greg KH
So, don't just wave it all away like this, it's a non-trivial
problem for a small number of systems that people are not
ignoring...

The developers of core system software should be forced to use 10
year old hardware to test their creation in daily use ;-)

[Claudio Freire]

Post by Ruediger Meier

Post by Stefan Seyfried

Post by Greg KH
Including the fact that most of us who have looked at it, can't
even duplicate the issue at all (myself included.)

Just try to use it on old (slow) rotating rust.
100% reproducible on all my 2.5" IDE HDDs here.

BTW my problem does not looked like HD speed issue. journalctl was
constantly on 100% CPU. So I assume that it couldn't handle more input
data in the same time (unless it's even more broken than thought).

Oh, about that, taking a look at perf-top could also give a hint.

[Greg KH]

Post by Stefan Seyfried

Post by Greg KH
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)

Just try to use it on old (slow) rotating rust.
100% reproducible on all my 2.5" IDE HDDs here.
shake -vvv shows thousands of fragments, probably due to lots of holes
in the database files.

What filesystem are you using? What is the % used of that partition?

I did try this on rust, but not a really slow one, I'll dig up an old
disk and try that next time I can.

Post by Stefan Seyfried

Post by Greg KH
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

The developers of core system software should be forced to use 10 year
old hardware to test their creation in daily use ;-)

Then nothing would ever get written because we would all be taking days
to build our kernels...

And you wouldn't get new features for that new hardware (transactional
memory, huge memory sizes, large numbers of cpus, etc.) You can't have
it both ways, sorry.

If you want, just run 10 year old software on that hardware, it should
work just fine.

greg k-h

[Stefan Seyfried]

Post by Greg KH

Post by Stefan Seyfried
shake -vvv shows thousands of fragments, probably due to lots of holes
in the database files.

What filesystem are you using? What is the % used of that partition?

ext3 and ext4, between 50% to 90% full.

Post by Greg KH
I did try this on rust, but not a really slow one, I'll dig up an old
disk and try that next time I can.

Post by Stefan Seyfried

Post by Greg KH
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

The developers of core system software should be forced to use 10 year
old hardware to test their creation in daily use ;-)

Then nothing would ever get written because we would all be taking days
to build our kernels...

compiling is allowed on current hardware ;-)

Post by Greg KH
And you wouldn't get new features for that new hardware (transactional
memory, huge memory sizes, large numbers of cpus, etc.) You can't have
it both ways, sorry.
If you want, just run 10 year old software on that hardware, it should
work just fine.

I'll probably just go for busybox-init/mdev instead of systemd/udev on
those machines -- boots exactly as fast but is much less demanding on
the hardware (and on admin-maintainance). And the feature set is good
enough.

My work machine has now gotten a SSD due to the "systemctl status
foo.service" taking ages (and "git status" in kernel trees, too)

For my wife's old machine, I recently bought an IDE-SSD to get it to
boot in an acceptabe time frame with current 13.1. That's of course not
only systemd's fault, the whole userspace stuff is getting much fatter
all the time.

Have fun,

seife

[Greg KH]

Post by Stefan Seyfried

Post by Greg KH

Post by Stefan Seyfried
shake -vvv shows thousands of fragments, probably due to lots of holes
in the database files.

What filesystem are you using? What is the % used of that partition?

ext3 and ext4, between 50% to 90% full.

Ok, that helps, both of these, when things start to get full start
thrashing in well-known ways to find free blocks.

thanks,

greg k-h

[Jan Engelhardt]

Post by Stefan Seyfried
My work machine has now gotten a SSD due to the "systemctl status
foo.service" taking ages (and "git status" in kernel trees, too)
For my wife's old machine, I recently bought an IDE-SSD to get it to
boot in an acceptabe time frame with current 13.1. That's of course not
only systemd's fault, the whole userspace stuff is getting much fatter
all the time.

Yeah, let's go back to the DOS times, where there was just 60 or so
programs in C:\DOS (something like /bin). Tab completion would
complete in a blink — man, where have we developed toward.

Well, with SYSV scripts going away thanks to systemd, you save the
overhead of interpreting (repetitive) sh code. So now, your boot
phase should be more IO-bound, and SSD is a good investment in that
regard.

[Carlos E. R.]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Greg KH

Post by Carlos E. R.

Post by Stefan Seyfried

Post by Ruediger Meier
That's only 4K/s. Is this normal?

Let me guess: journal on a normal (non-ssd) disk?
Obviously it is not designed for such outdated hardware ;-)
The file fragmentation of the journal files and the performance of the
"database" is absolutely totally horrible.
It gets slightly better once you dedicate a fast SSD to the journal ;)

Bad programming, IMHO.

Really? Patches are always gladly accepted upstream for this, and
numerous people have looked at it to try to firstly determine what
exactly the problem is, and no one has been able to figure that out.
Including the fact that most of us who have looked at it, can't even
duplicate the issue at all (myself included.)
So, don't just wave it all away like this, it's a non-trivial problem
for a small number of systems that people are not ignoring...

Maybe you need expert database engine programmers in the team. I'm not.


If programmers are used to excellent and modern hardware, you can not make
code that works well with the kind of hardware normal people use. This
happened to me, as a programmer... it is not idle talk.

Remember that one of the sale points of Linux was that it brings new life
to older hardware that no longer works well with the last Windows version.
If you insist that openSUSE must run on SSD media, you are breaking
contact with the masses of Linux users out there.

- --
Cheers,
Carlos E. R.
(from 12.3 x86_64 "Dartmouth" at Telcontar)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)

iEYEARECAAYFAlJ1eBMACgkQtTMYHG2NR9UxrACfRAKyrXZ9YjXhQXXNJN2S5ROX
O/cAni4CA69vw8v12mJAYbwO6MkwSpoA
=4DOU
-----END PGP SIGNATURE-----

[Bruno Friedmann]

Post by Ruediger Meier
Hi,
I'am trying to watch the logs but it's a bit too slow.
journalctl
is running since 8 hours now and it has printed only 120MB text yet.
That's only 4K/s. Is this normal?
journalctl also consumes 100% CPU the whole time so the machine is
useless currently.
cu,
Rudi

/me just loving journald + systemd-logger

journalctl is a perfect tool excepted it slowlyness (even on ssd) but you certainly don't want more than since the last boot
so using -b fix for 98% my use case.

syntax color, search, bold, export in text file etc ...

Thanks to have invented that tool!
--
Bruno Friedmann
Ioda-Net Sàrl www.ioda-net.ch

openSUSE Member
GPG KEY : D5C9B751C4653227
irc: tigerfoot