Centralized_Logging

Centralized_Logging#

Motivation#

When a system grows to multiple hosts, managing the logs and accessing them can get complicated. Searching for a particular error across hundreds of log files on hundreds of servers is difficult without good tools. FreeIPA currently does not have documentation or even own tools how to set up the infrastructure and it is being solved separately by each FreeIPA user. This page summarizes the efforts and development of a Centralized Logging solution in/for FreeIPA that would close that gap.

FreeIPA is a security related project, usually a backbone of the infrastructure. Tracking, audit or analysis of the logs of the FreeIPA server behavior and related FreeIPA user behavior is useful as it allows detecting problems in time before they grow in even bigger problems (e.g., replication errors) or helps detecting attacks, misbehavior or simply configuration problems in the infrastructure and amending them.

FreeIPA logs#

FreeIPA server itself is not a monolithic application, but rather compound of selected open source projects creating a solution for identity management. This makes log collection, processing and aggregation a challenge as each project have own way and technology how to structure and produce logs. This applies both for FreeIPA server and it’s many services, but also for FreeIPA clients, where different information lies in different logs (SSSD log, audit log).

Use Cases#

Before designing a particular solution, basic use cases and intents of what logs will be used and how should be defined. Currently, following set of use cases is tracked:

  • As Admin or Auditor, I want to see all calls to FreeIPA API so that I can audit administrative changes to FreeIPA server (for instance: who added user “foo”? Who revoked a certificate and when?)

  • As Auditor, I want to see all logins attempts for meeting security requirements and forensic analysis

  • As Security Administrator, I want to visualize failed login attempts to brute-force attack

  • As Admin, I want to see replication status of all my FreeIPA replicas so that I can amend the issue in a timely manner to avoid replication conflicts or out-of-sync data (I want my login to work on all servers)

  • As Admin, I want to identify AD interoperability problems on FreeIPA server (example: SSSD on the FreeIPA server cannot talk to the Active Directory server and thus resolve groups)

  • As Admin, I want to see load of Kerberos and LDAP server to be able to compare the load on the FreeIPA servers (and for example identify badly configured DNS SRV records or clients always calling the same server), network problems (some servers may not be reachable by most clients and have thus very small load) or simply by some of the services not running.

Proposed Solution#

The centralized log collection problem has 2 basic parts:

  • Selecting and configuring the right centralized server that will collect, process, annotate and visualize the logs

  • Configuring the actual machines (FreeIPA servers or clients) to send the interesting logs to the centralized server and selecting proper and secure transport for the logs.

Centralized Logging Server#

Currently, the most used and advanced open source solutions are the centralized servers based on Elasticsearch for data storage and searches, rsyslog/logstash/fluentd for log reception and processing and Kibana for data display and dashboards (commonly referred to as REK, ELK servers based on the log receiving part).

Currently proposed solution chosen rsyslog. The server is packaged as Docker image. See the Docker Hub Registry or respective github repo for details how to download and run the preconfigured REK server.

Log Gathering and Transport#

To meet the defined use cases, at least following logs have to be gathered from the FreeIPA server:

  • /var/log/httpd/error_log: FreeIPA API call logs (and Apache errors)

  • /var/log/krb5kdc.log: FreeIPA KDC utilization

  • /var/log/dirsrv/slapd-$REALM/access: Directory Server utilization

  • /var/log/dirsrv/slapd-$REALM/errors: Directory Server errors (including mentioned replication errors)

  • /var/log/pki/pki-tomcat/ca/transactions: FreeIPA PKI transactions/logs


FreeIPA client logs:

  • /var/log/sssd/*.log: SSSD logs (multiple, for all tracked logs)

  • /var/log/audit/audit.log: user login attempts

  • /var/log/secure: reasons why user login failed


For easier FreeIPA log forwarding configuration, ipa-log-config tool can be used. Visit its github page for more information and usage.

Results#

Current the proof of concept project contains:

  • Aforementioned CentOS-7 based REK server for log collection (Docker image)

  • Set of preconfigured Kibana dashboards visualizing different logs in readable graphs. See the next subsection for details. (Kibana dashboards and searches)

  • Basic log message filtering rules for the rsyslog on the server, turning the log messages into structured logs that are searchable in Kibana (rsyslog rules)

  • Script to configure rsyslog-based log transmission on the FreeIPA servers and clients (github page)

Dashboards#

Demo of the Kibana Dashboards#

{{#ev:youtube|7YjA6z5nE0I|640|center}}

Other Resources#