3KeyPKITechnical

SignServer Dashboards and Reporting

Our post regarding EJBCA Dashboard and Reporting has a really great feedback, people are discussing with us how to make efficient visualisations to be able to read important information on the fly, or how it can be adjusted to provide monitoring capabilities. There are many more possibilities how to use it to support your business case.

Based on the positive feedback we have decided to write a few words about how to use the same technology in order to see information coming from SignServer. Who knows EJBCA, SignServer is a great solution for centralised signing operations. It has a lot of benefits as we can enforce security policy on the server which can be applied for different signature format and types, or we can define our own custom signing output format besides the standard ones like PAdES, XAdES, CAdES, Authenticode, Timestamps, etc. If you are not aware of SignServer, I would recommend you to have a look at it.

So what are some basic information we would like to see from the signing perspective? Few examples are:

  • Overall distribution of requests coming to SignServer and processed
  • Activities performed by identified administrators of the SignServer
  • Ratio between successful and failed requests
  • Distribution of cryptographic keys used for signing and related CryptoTokens
  • Map of the requests coming around the globe
  • Workers overall count of requests coming in time

As you can imagine there are many more use cases we can implement and correlate. Using the power of ELK stack we can provide on top of that many additional features like monitoring, reporting, alerting, or machine learning to predict the behaviour and performance of SignServer nodes in time. All of that can be linked to procedures and provide automation.

High level overview

The setup is similar to the one used in case of EJBCA. This is caused by the similar core components used by both SW products. However in case of SignServer there are differences on how are services provided and consumed, which has impact on the data we can use.

The following setup is used for the demonstration purposes:

  • SignServer signing node
  • PKCS#11 and PKCS#12 CryptoToken
  • PDFSigner used to sign PDF files with the cryptographic key in PKCS#11 CryptoToken
  • DemoTimeStampSigner producing digital timestamps using the key stored in PKCS#12 CryptoToken
  • Wildfly as application server running the SignServer and configured to ship logs to ELK stack
  • ELK stack consisting of Logstash parsing the syslog data, Elasticsearch to store index data, and Kibana to provide visualisation, dashboarding, and other relevant features from the web interface

For the integration purposes, there is no need to develop any custom code. Application server is taking care of shipping the logs through the syslog interface to Logstash, where all the magic and parsing happen.

Logs produced by SignServer

Logging of SignServer uses some common components with EJBCA however it has a different structure. Documentation can be found on the official web site, but it is not very accurate about the format and we cannot rely on it during the parsing of logs. So we need to stick with the empirical analysis of what is going on during the particular operations of the SignServer or use the source code to see how exactly are logs produced.

We will split logs into three different categories:

  • System logs – produced by the system and containing information about the application operations
  • Admin logs – produced by the administrator operations
  • Worker logs – produced by workers when there is an operation processed

As an example we can take the admin log entry as follows:

2020-03-20 18:26:01,523 INFO  [org.signserver.admin.common.auth.AdminAuthHelper] (default task-43) ADMIN OPERATION; subjectDN=O=3Key Company s.r.o., CN=SuperAdmin; serialNumber=3b886838ee3611001b0cbf06d2d2ebc83af59d4; issuerDN=O=3Key Company s.r.o., CN=Management CA; authorized=true; operation=getStatus; arguments=Worker{id: 110},;    

By default, it has the standard format we can parse using the following regular expresion:

%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n

However, the most important part with the relevant information we would like to use is the message contained in last part of the log:

ADMIN OPERATION; subjectDN=O=3Key Company s.r.o., CN=SuperAdmin; serialNumber=3b886838ee3611001b0cbf06d2d2ebc83af59d4; issuerDN=O=3Key Company s.r.o., CN=Management CA; authorized=true; operation=getStatus; arguments=Worker{id: 110},;

As you can see, it contains the identification of administrator operations at the beginning, and the rest of the message are key-value pairs, containing key as identification of the message attribute. This can be used to simplify our Logstash configuration later.

Similar approach can be used for system and worker logs. It will require few changes as we need to identify the type of the log and its format and structure.

Processing logs

When we have the understanding of logs and its formats, we can define the Logstash pipeline we will use to process logs coming from the application server. Pipeline has the standard parts: input, filter, and output.

Input

There is nothing special about the input part. It will create a listener and provide a basic definition of what is expected to come, in this case syslog message, which can contain multiline logs, such as Java Exceptions.

input {
  syslog {
    host => "0.0.0.0"
    port => 5144

    codec => multiline {
      pattern => "<%{POSINT:priority}>%{SYSLOGLINE}"
      negate => "true"
      what => "previous"

      auto_flush_interval => 1
      max_lines => 500
      charset => "UTF-8"
   }
  }
}

Filter

This is the most important part where we are parsing logs coming form application server. As a first step we will add some fields and rename syslog for better structure:

  mutate {
    add_field => {
      "logstash_node" => "lab04.3key.company"
      "[@metadata][index]" => "signserver-log"
    }

    rename => {
      "message" => "original_message"
      "logsource" => "[hostname]"
      "facility" => "[syslog][facility]"
      "facility_label" => "[syslog][facility_label]"
      "severity" => "[syslog][severity]"
      "severity_label" => "[syslog][severity_label]"
      "priority" => "[syslog][priority]"
      "pid" => "[syslog][pid]"
      "program" => "[syslog][program]"
    }
  }

The message received is contained in the “original message” field. We can use the grok pattern to get the separate filed from it as stated above in the part about the format of SignServer logs. Then based on the logger type and the initial part of the [signserver][message], we can  identify administrator operation and parse it accordingly.

  # parse syslog and signserver message
  grok {
    match => { "original_message" => "" }
  }

  if "_grokparsefailure" not in [tags]{
    mutate {
      gsub => [
        # Remove whitespaces from string like "default-threads - 24" if contain dash
        "[ejbca][thread]", " - ", "-",
        # Replace the spaces with a dash
        "[ejbca][thread]", " ", "-"
      ]
      remove_field => [ "original_message", "host", "timestamp" ]

    }
  }

  # ADMIN_OPERATION message like:
  # ADMIN OPERATION; subjectDN=O=3Key Company s.r.o., CN=SuperAdmin; serialNumber=3b886838ee3611001b0cbf06d2d2ebc83af59d4;
  # issuerDN=O=3Key Company s.r.o., CN=Management CA; authorized=true; operation=getStatus; arguments=Worker{id: 2},;
  #
  if [signserver][message] =~ /^ADMIN OPERATION;/ {
    grok {
      match => { "" }
    }

    mutate {
      replace => { "[@metadata][index]" => "signserver-admin" }
    }
  }

And finally, we can get all the key-value pairs from the administrator operation message, which will create additional fields in the index based in the content of the message:

    kv {
      allow_duplicate_values => false
      field_split_pattern => "; "
      source => "[signserver][adminoperation_message]"
      target => "[signserver][adminoperation]"
      transform_key => "lowercase"
    }

We can apply the similar approach also for worker and system logs in order to get all relevant information into fields that can be searchable.

Output

As a last part we need to index the data into Elasticsearch. This is simple as we already have all the data and fields prepared at this stage:

output {
    elasticsearch {
      hosts => ["http://127.0.0.1:9200"]
      index => "%{[@metadata][index]}-%{+YYYY.MM.dd}"

      user => "logstash_signserver_user"
      password => ""
    }
}

Configure shipping of logs

When the pipeline to parse incoming syslog messages on Logstash is prepared, we can configure Wildlfy to ship logs to its destination. We can configure what logs we would like to ship based on the logger definition of category and level. In our example, we are shipping all logs which has INFO level and categories org.signserver and org.cesecore.

<custom-handler name="SYSLOG" class="org.jboss.logmanager.handlers.SyslogHandler" module="org.jboss.logmanager">
    <level name="INFO"/>
    <properties>
        <property name="serverHostname" value="lab04.3key.company"/>
        <property name="hostname" value="lab01.3key.company"/>
        <property name="port" value="5144"/>
        <property name="protocol" value="TCP"/>
        <property name="appName" value="signserver"/>
        <property name="facility" value="LOCAL_USE_7"/>
        <property name="encoding" value="US-ASCII"/>
        <property name="syslogType" value="RFC3164"/>
        <property name="maxLength" value="65000"/>
    </properties>
</custom-handler>

<logger category="org.signserver">
    <level name="INFO"/>
    <handlers>
        <handler name="SYSLOG"/>
    </handlers>
</logger>

<logger category="org.cesecore">
    <level name="INFO"/>
    <handlers>
        <handler name="SYSLOG"/>
    </handlers>
</logger>

Mapping, visualisation and dashboard

When all configurations are correct, we should start to see the information in index documents through the Kibana. If not, there may be some parsing error and you should try to identify any issue through the Logstash or Elasticsearch.

Now, we are ready to create different visualisations based on these data. We can, of course, adjust the mapping of the index, create a mapping template of the index, to provide more precise definition of the data types which we would like to use for aggregations (for example to show the incoming request on the map).

As an example of simple visualisation can be:

  • Administrator operations distribution
  • Most active administrators
  • Total count of activity in time
  • System events distribution
  • Timestamping status ratio
  • Worker counts of processed requests
  • CryptoToken and related cryptographic key usage distribution
  • Most active workers
  • Global map of the incoming worker requests

We can then put visualisations in a dashboard which can be very useful tool for the purpose of monitoring or getting the current information about the status of the SignServer workers.

This is very handy for all types of teams to see what is going on without looking and searching through a bunch of different logs in different systems. 

Moreover, correlation with other data can create really nice and interesting use cases that can be handled easily (definition of workflow that can be triggered based on reaching the threshold or learning the baseline of the operations and proactively react on the upcoming events) and therefore reduce the time needed by the operational team to manage the solution.

Get in touch with us to know more!

You can find the configuration and setup code on the GitHub page.

We are also on LinkedIn, follow us!

If you like what we are doing than do not hesitate and contact us for more information! We believe that we have experience and knowledge that can be a benefit for your business.