Discussion
Loading...

Discussion

  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
Marc
@corpsmoderne@mamot.fr  ·  activity timestamp 2 months ago

... or maybe can someone explain to me the difference between #elasticsearch apm and esc? Are they 2 completely separate systems or can I use apm to generate ESC logs?

  • Copy link
  • Flag this post
  • Block
Marc
@corpsmoderne@mamot.fr replied  ·  activity timestamp 2 months ago

So in the end using a channel like it's done in the console-subscriber works: https://github.com/tokio-rs/console/blob/main/console-subscriber/src/lib.rs#L440-L475

  • Copy link
  • Flag this comment
  • Block
Epic Eric :thinkhappy:
@epiceric@mastodon.xyz replied  ·  activity timestamp 2 months ago
@corpsmoderne Could you maybe use an unbounded mpsc channel?
  • Copy link
  • Flag this comment
  • Block
Marc
@corpsmoderne@mamot.fr replied  ·  activity timestamp 2 months ago

... or maybe can someone explain to me the difference between #elasticsearch apm and esc? Are they 2 completely separate systems or can I use apm to generate ESC logs?

  • Copy link
  • Flag this comment
  • Block
Toasterson
@Toasterson@chaos.social replied  ·  activity timestamp 2 months ago
@corpsmoderne i have elasticsearch at $WORK and admin it. APM is purely to capture performance data. Not needed for logging at all. Simplest and most robust way to get logs to es is. Use filebeat. On kubernetes you can emit any JSON log lines to stdout and have filebeat consume and sent it. Filebeat deals with es being offline and other edge cases. In many cases when our elastic server had a hickup all our apps died. So filebeat is heartly recomended.
  • Copy link
  • Flag this comment
  • Block
Marc
@corpsmoderne@mamot.fr replied  ·  activity timestamp 2 months ago
@Toasterson ... and now I understand why there's a tracing_ecs crate which prints ESC jsons to stdout 😅 . Well, I've my own ECS tracing layer now.
  • Copy link
  • Flag this comment
  • Block
Toasterson
@Toasterson@chaos.social replied  ·  activity timestamp 2 months ago
@corpsmoderne A fun side note should you ever go down the Filebeat Route is: Filebeat adds it's own ecs logging to the final elasticsearch logs thus duplicates the metadata. It's simpler to print JSON and then have filebeat consume that.
  • Copy link
  • Flag this comment
  • Block
Marc
@corpsmoderne@mamot.fr replied  ·  activity timestamp 2 months ago

Yeah now I've a purely #elasticsearch #esc issue, When I try to send a new event the call fails with a 500 and cryptic error, and google is not helpful...

STATUS: 500, BODY: {
"error": {
"reason": "[_data_stream_timestamp] meta field has been disabled",
"root_cause": [
{
"reason": "[_data_stream_timestamp] meta field has been disabled",
"type": "illegal_state_exception"
}
],
"type": "illegal_state_exception"
},
"status": 500
}

  • Copy link
  • Flag this comment
  • Block
Toasterson
@Toasterson@chaos.social replied  ·  activity timestamp 2 months ago
@corpsmoderne strip the timestamp from your json. Looks like the target datastream is setup to populate the timestamp itself.
  • Copy link
  • Flag this comment
  • Block
Marc
@corpsmoderne@mamot.fr replied  ·  activity timestamp 2 months ago
@Toasterson ah thanks but I managed to make it work, I was sending to a wrong index it seems. The timestamp was needed.
  • Copy link
  • Flag this comment
  • Block
Toasterson
@Toasterson@chaos.social replied  ·  activity timestamp 2 months ago
@corpsmoderne Ah, yeah thats also a common one, when you send to the wrong index which does not like your field mapping.
  • Copy link
  • Flag this comment
  • Block
Log in

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.0-rc.3.1 no JS en
Automatic federation enabled
  • Explore
  • About
  • Members
  • Code of Conduct
Home
Login