2

Good day;

I need to test my log management stack's database query performance and in order to do so, I need to generate bulks of log messages that seem to be six months old and send them to rsyslogd.

Is this even possible? If yes please tell me how.

Apologies in advance if anything's wrong with this question, I'm a newbie.

I really appreciate any help you can provide.

3
  • Would it count to just write logs with old timestamps directly to the log file? Jul 8, 2022 at 15:25
  • Hello @Questionmark, since writing to /var/log/syslog would be as same as sending logs to rsyslogd socket, that might actually work.
    – Sinux
    Jul 9, 2022 at 6:52
  • Hello again, I've tested this and @thrig is right, rsyslogd doesn't preserve the timestamp and overrides it.
    – Sinux
    Jul 9, 2022 at 14:12

1 Answer 1

2

Bulk Database Load

One way is to bulk load records into the logging system database. This could be easier that having to generate custom syslog records and hoping that the timestamp of the sender is preserved, and not the one invented by the syslog daemon. Also database bulk loads should be much more efficient than sending and parsing of a lot of syslog messages.

Downsides: there may not be direct database access, a bad upload could drop or trample existing records, etc.

API Upload

The logging system may have an API. This will likely be less efficient than a direct database upload, but does avoid the risk of dropping the entire database. Consult the documentation for the logging system for how this needs to be used (if it exists).

Native Syslog Send

Otherwise [RFC 3164] observes the syslog protocol in its native habitat; there may be ready-made code available though such code may not offer the ability to forge the timestamp, depending on the implementation (neither logger(1) nor syslog(3) offer an obvious way to forge the timestamp, nor does the Net::Syslog Perl module, etc). On the other hand it's not too hard to forge messages with a custom timestamp:

#include <sys/socket.h>
#include <sys/types.h>

#include <netdb.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

#define SYSLOGMAX 1024

char buf[SYSLOGMAX];
struct addrinfo hints;

int mkcon(const char *host, const char *serv, const struct addrinfo *hints);

int
main(void)
{
    hints.ai_socktype = SOCK_DGRAM;
    int server        = mkcon("192.168.99.101", "syslog", &hints);

    for (size_t i = 0; i < 4; i++) {
        // this is the older [RFC 3164] form, which is simpler
        sprintf(buf, "<%d>Jan 11 12:13:14 host program[123]: blah%lu",
                (21 << 3) | 4, i);

        send(server, buf, SYSLOGMAX, 0);
        // network admins or logging services might get unhappy
        // if the packets or log messages show up too quick
        usleep(100000);
    }

    return 0;
}

int
mkcon(const char *host, const char *serv, const struct addrinfo *hints)
{
    struct addrinfo *peer;

    // TODO needs better error handling and better use of the
    // peer struct, etc
    int ret = getaddrinfo(host, serv, hints, &peer);
    if (ret != 0) abort();

    int server =
      socket(peer->ai_family, peer->ai_socktype, peer->ai_protocol);
    if (server == -1) abort();

    if (connect(server, peer->ai_addr, peer->ai_addrlen) == -1) abort();
    freeaddrinfo(peer);

    return server;
}

You may be able to point this at /dev/log which likely would be more efficient and less lossy than lobbing packets over the network. Modern syslog daemon may also support more reliable TCP streams. Check the documentation or relevant RFC for how that works.

4
  • Hello, if I've understood correctly from your answer, one way is to write these messages directly into the database, and another way is using the code you've kindly provided and point it to /dev/log to generate these messages, right? I'm currently testing them both but had to make sure I've understood your answer.
    – Sinux
    Jul 9, 2022 at 7:02
  • There are three methods, first bulk load the database, second generate packets using an internet socket address (shown above), third generate packets going to /dev/log (not shown above). Modern syslog daemons also support TCP but that's more work for a client, so for that finding a library or module that lets you forge the date would likely be the way to go.
    – thrig
    Jul 9, 2022 at 13:59
  • apparently the best approach is to bulk load the database. I can do that via Loki HTTP API, however, I haven't been able to generate a valid RFC5424 log yet, any ideas? thanks in advance.
    – Sinux
    Jul 9, 2022 at 14:14
  • The bulk loading the database works just fine and you were right it really is the most efficient method. Thank you so much and lots of love your way.
    – Sinux
    Jul 9, 2022 at 14:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .