This is our recipe for logging. We've used it in many projects and it works really well:
• Create an Events interface and inject it as a Hexagonal port into your application.
• Use EventsFilter to attach metadata such as traces and timestamps and event names (and this way you will get consistent field naming)
• Start small and consistent - create a logging library if polyrepo, and roll it out to the services to get a consistent logging format, migrating over current log messages, before finally deleting the logging library entirely
• Ban all unstructured logging - use Events only and transform into JSON reflectively or with generated adapters
• Use types for everything - attempts to log raw strings without a type wrapper fail. This will allow you to control the outflow of sensitive data (eg. PII). You can use values4k or similar to control masking of what goes out. By using a custom JSON marshaller then you can also ban PII types from even being logged entirely.
• No log levels at all. If you need these to see what is going on then you have kind of failed :)
• Only log what is interesting/relevant - if all HTTP traffic is too much then just filter for NON-200 traffic
• We go to Standard Out (but this can be injected with different implementations if required including for testing)
• Use kibana/cloudwatch etc to query the data consistently and build up dashboard and queries which can be reused.
Just do it, then show people what they're missing! 🙂 . We will be showing what kinds of thing are possible at our KotlinConf presentation.