Hey rustaceans, I need to plug my #axum server which uses the tracing crate for logs to our elasticsearch logging service, which uses ESC I believe?
It seems there's no off the shelf tracing_layer to talk to elasticsearch , and trying to write my own I'm running in the issue of needing to use an async method to send the message from a sync context (on_event() of tracing::Layer). Anything I've missed to make my live easier? #rustlang #tokio #elasticsearch
Discussion
Loading...
So in the end using a channel like it's done in the console-subscriber works: https://github.com/tokio-rs/console/blob/main/console-subscriber/src/lib.rs#L440-L475
@corpsmoderne Could you maybe use an unbounded mpsc channel?
... or maybe can someone explain to me the difference between #elasticsearch apm and esc? Are they 2 completely separate systems or can I use apm to generate ESC logs?
@corpsmoderne i have elasticsearch at $WORK and admin it. APM is purely to capture performance data. Not needed for logging at all. Simplest and most robust way to get logs to es is. Use filebeat. On kubernetes you can emit any JSON log lines to stdout and have filebeat consume and sent it. Filebeat deals with es being offline and other edge cases. In many cases when our elastic server had a hickup all our apps died. So filebeat is heartly recomended.
@Toasterson ... and now I understand why there's a tracing_ecs crate which prints ESC jsons to stdout 😅 . Well, I've my own ECS tracing layer now.
@corpsmoderne A fun side note should you ever go down the Filebeat Route is: Filebeat adds it's own ecs logging to the final elasticsearch logs thus duplicates the metadata. It's simpler to print JSON and then have filebeat consume that.
Yeah now I've a purely #elasticsearch #esc issue, When I try to send a new event the call fails with a 500 and cryptic error, and google is not helpful...
STATUS: 500, BODY: {
"error": {
"reason": "[_data_stream_timestamp] meta field has been disabled",
"root_cause": [
{
"reason": "[_data_stream_timestamp] meta field has been disabled",
"type": "illegal_state_exception"
}
],
"type": "illegal_state_exception"
},
"status": 500
}
@corpsmoderne strip the timestamp from your json. Looks like the target datastream is setup to populate the timestamp itself.
@Toasterson ah thanks but I managed to make it work, I was sending to a wrong index it seems. The timestamp was needed.
@corpsmoderne Ah, yeah thats also a common one, when you send to the wrong index which does not like your field mapping.
bonfire.cafe
A space for Bonfire maintainers and contributors to communicate
Automatic federation enabled