Hacker Newsnew | past | comments | ask | show | jobs | submit | valyala's commentslogin

Coroot is the future of observability

Could you say more about your experience and what you like about it?

It monitors all the applications in your network and automatically detects the most common issues with the applications. It also collects metrics, traces, logs and CPU profiles for the monitored applications, so you could quickly investigate the root cause of various issues if needed.

I like that Coroot works out of the box without the need in complicated configuration.


The number of active users at StackOverflow started dropping in the middle of 2020, i.e. long time before ChatGPT release in the end of 2022.

https://data.stackexchange.com/stackoverflow/revision/192836...


Nice article about the usefulness of wide events! It's pity it doesn't name open-source solutions optimized for wide events such as VictoriaLogs.


Try open-source databases specially designed for traces, such as Grafana Tempo or VictoriaTraces. They can handle the data ingestion rate of hundreds of thousands trace spans per second on a regular laptop.


What is the purpose of MinIO, Seaweedfs and similar object storage systems? They lack durability guarantees provided by S3 and GCS. They lack "infinite" storage promise contrary to S3 and GCS. They lack "infinite" bandwidth unlike S3 and GCS. They are more expensive than other storage options, unlike S3 and GCS.


We use it because we are already running our own k8s clusters in our datacenters, and we have large storage requirements for tools that have native S3 integration, and running our own minio clusters in the same datacenter as the tools that generate and consume that data is a lot faster and cheaper than using S3.

For example, we were running a 20 node k8s cluster for our Cortex (distributed Prometheus) install, monitoring about 30k servers around the world, and it was generating a bit over a TB of data a day. It was a lot more cost effective and performant to create a minio cluster for that data than to use S3.

Also, you can get durability with minio with multi cluster replication.


Consider migrating to VictoriaMetrics and saving on storage costs and operations costs. You also won't need MinIO, since it stores data to local filesystem (aka to regular persistent volumes). See real-world reports from happy users who saved costs on a large-scale Prometheus-compatible monitoring - https://docs.victoriametrics.com/victoriametrics/casestudies...


I can't imagine switching at this point. We spent quite a while building up our Cortex and Minio infrastructure management, as well as our alerting and inventory management systems, and it is all very stable right now. We don't really touch it anymore, it just hums along.

We have already worked through all the pain points and it all works smoothly. No reason to change something that isn't a problem.


I haven't used it in a while, but it used to be great as a test double for s3


S3 is a widely supported API schema, so if you need something on-prem, you use these.


But what's the point to use these DIY object storage systems, when they do not provide durability and other important guarantees provided by S3 and GCS?


When you want just the API for compatibility, I guess?

Self-hosted S3 clones with actual durability guarantees exist, but the only properly engineered open source choices are Ceph + radosgw (single-region, though) or Garage (global replication based on last-writer-wins CRDS conflict resolution).


It's great for a prototype which doesn't need to store a huge amount of data, you can run it on the same VM as a node server behind Cloudflare and get a fairly reliable setup going


Minio allows you to have an s3 like interface when you have your own servers and storage.


MinIO also allows losing your data, since it doesn't provide high durability guarantees unlike S3 and GCS.


> we did, I think the entire journey was 7 years long, communicated many times, over at least 6 major releases. maintaining dashboards in two languages increased complexity, whilst reducing compatibility, and gave a very large security surface to be worried about. we communicated clearly, provided migration tools, put it in release notes, updated docs, repeated it at conferences and on community calls

If this migration appeared to be so painful, why did you decide to finish it (and make users unhappy) instead of cancelling the migration at early stages? What are benefits of this migration?


> many are adopting to gain independence from vendors

This is the worst reason to migrate to OTEL format for metrics, since every vendor and every solution for metrics has its' own set of transformation rules for the ingested OTEL metrics before saving them into the internal storage (this is needed in order to align OTEL metrics to the internal data model unique per each vendor / service). These transformation rules are incompatible among vendors and services. Also, every vendor / service may have its own querying API. This means that users cannot easily migrate from one vendor / service to another one by just switching from the old format to OTEL format for metrics' transfer. Read more about this at https://x.com/valyala/status/1982079042355343400


Boring monitoring software is rare, since it is hard to refuse adding yet another shiny feature (and breaking the core functionality along the way) instead of focusing on improving the usability for the core functionality.

We at VictoriaMetrics try making boring monitoring solution which just works out of the box - https://docs.victoriametrics.com/victoriametrics/goals/



You can use other monitoring systems, which support both pull-based and push-based models for metrics' collection. See, for example, https://docs.victoriametrics.com/victoriametrics/#how-to-imp...


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: