It looks neat though, but I won't burn myself on anything closed source if there's open source and/or self-hosted alternatives.
tschellenbach 14 minutes ago [-]
Doesn't always help. Price hikes on CockroachDB for instance have been crazy.
mehrant 18 minutes ago [-]
thanks for taking time and commenting :) we'd still be happy if you decided to use it and give us your thoughts. as I mentioned in one of the comments below, we're hoping to go open source in future :)
verdverm 4 minutes ago [-]
> we're hoping to go open source in future :)
That's just lip service, either you intend to or you don't, it's not up to hope
We hope you will see the dev tooling space is based around open source and you will alienate many potential users by not being open source
Snawoot 53 minutes ago [-]
> 2-6x faster than Redis (benchmark link below) yet disk persisted!
That's a false contradistinction: Redis is also disk persisted.
The benchmark you did mentions Redis benchmarking guide and this guide has following paragraph:
> Redis is, mostly, a single-threaded server from the POV of commands execution (actually modern versions of Redis use threads for different things). It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store.
Did you just benchmarked against only single Redis instance and claimed performance win? Even if so, how do benchmarks compare against source-available competitor DragonflyDB?
Finally, documentation doesn't mention how persistence exactly works and what durability guarantees should we expect?
mehrant 35 minutes ago [-]
thanks for taking time to write a feedback :)
> That's a false contradistinction: Redis is also disk persisted.
The performance gain mentioned was vs. Redis in memory. so we weren't claiming that Redis can't be persisted (which of course it can), but we were saying that Redis without persistence (which performs faster that with persistence) was still this much slower than HPKV with persistence.
But you're correct that we probably should have been more clear in explaining this :)
>Did you just benchmarked against only single Redis instance and claimed performance win?
Signle node of Redis vs. Single node of HPKV. so it's an apples to apples comparison
>Even if so, how do benchmarks compare against source-available competitor DragonflyDB?
Benchmark with DragonFly coming soon :)
sorry about lack of that information in documentation, we'll update that. for for now, the durability guarantee on Pro is 30 seconds. on Business with HA is 5 minutes.
alex_smart 41 minutes ago [-]
I don’t get it. How could you be fsyncing the WAL in 600ns? What are the transactional guarantees that you are offering?
mehrant 20 minutes ago [-]
that's a great question. the 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through:
1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers.
2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge.
3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers
the transactional guarantees we provide is via:
1- atomic individual operations with retries
2- various conflict resolution strategies (timestamp, etc.)
3- durability through controlled persistence cycles with configurable commit intervals
depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
buenzlikoder 5 minutes ago [-]
What storage backend are you using?
A write operation on a SSD takes 10s of uS - without any VFS layers
dangoodmanUT 44 minutes ago [-]
What disks give 600ns persistence _with fsync/fdatasync_? Never heard of anything under 50us p50.
mehrant 19 minutes ago [-]
the 600ns figure represents our optimized write path and not a full fsync operation. we achieve it -among other things- through:
1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers.
2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge.
3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers
the transactional guarantees we provide is via:
1- atomic individual operations with retries
2- various conflict resolution strategies (timestamp, etc.)
3- durability through controlled persistence cycles with configurable commit intervals
depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
dangoodmanUT 2 minutes ago [-]
I didn't necessarily mean exactly fsync. I guess I'll ask: Is it actually flushed to persistent disk in 600ns such that if the node crashes, the data can always be read again? Or does that not fully flush?
quibono 43 minutes ago [-]
Is this 2-6x faster because of multi threading/core? Or is this actually 2-6x faster on a single core machine?
mehrant 19 minutes ago [-]
the test was done on a single node and a single thread. on multi thread and batch operations, HPKV was still faster on the same machine
kshmir 51 minutes ago [-]
Why pay what you're asking instead of using dragonfly or something like that and just putting a beefier node?
ehsanaslani 31 minutes ago [-]
Well that's a technical choice depending on the context, but I can list some of the advantages of HPKV:
-Persistent by default without any performance penalties
-The pub/sub feature which is unique to HPKV and allows for a bi-directional websocket connection from clients to database
-Lower cost as we need less expensive infrastructure to provide the same service
-Simple API to use
theonlyvasudev 47 minutes ago [-]
Amazing!
mehrant 19 minutes ago [-]
thank you :)
ranger_danger 1 hours ago [-]
Does not appear to be open-source.
mehrant 1 hours ago [-]
correct. however we're actually planning to make the system open source in future; we can't set an exact date as it depends on various factors, but hopefully not too far out. :)
notpushkin 1 hours ago [-]
Will be looking forward to that!
However, it feels a bit weird: at this level of performance going SaaS only kinda defeats the purpose, no?
mehrant 49 minutes ago [-]
our approach is actually hybrid. on the other side of the performance coin, we have resource efficiency. that resource efficiency let's us provide a performant and low latency managed KV store, with lower cost, so the economy of it makes sense. the idea is that not everyone requires sub-microsecond latency, and for that group the value proposition is a low latency kv store which is feature rich with a novel bi-directional ws api. for people who need sub-microsecond latency, we're planning custom setup that allows them to make a local vectored interface call to get the sub-microsecond speeds. in between, we have the business plan that provides a custom binary protocol that is the one used in the benchmark :)
huhtenberg 1 hours ago [-]
In that case you need to provide an SLA for your speed claims. Otherwise the claims are basically moot.
mehrant 46 minutes ago [-]
that's a fair point and you're correct. we will have the SLAs for latency documented and provided soon. in the mean time, please try it out and give us your feedback :)
56 minutes ago [-]
Rendered at 16:08:46 GMT+0000 (Coordinated Universal Time) with Vercel.
It looks neat though, but I won't burn myself on anything closed source if there's open source and/or self-hosted alternatives.
That's just lip service, either you intend to or you don't, it's not up to hope
We hope you will see the dev tooling space is based around open source and you will alienate many potential users by not being open source
That's a false contradistinction: Redis is also disk persisted.
The benchmark you did mentions Redis benchmarking guide and this guide has following paragraph:
> Redis is, mostly, a single-threaded server from the POV of commands execution (actually modern versions of Redis use threads for different things). It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store.
Did you just benchmarked against only single Redis instance and claimed performance win? Even if so, how do benchmarks compare against source-available competitor DragonflyDB?
Finally, documentation doesn't mention how persistence exactly works and what durability guarantees should we expect?
> That's a false contradistinction: Redis is also disk persisted.
The performance gain mentioned was vs. Redis in memory. so we weren't claiming that Redis can't be persisted (which of course it can), but we were saying that Redis without persistence (which performs faster that with persistence) was still this much slower than HPKV with persistence. But you're correct that we probably should have been more clear in explaining this :)
>Did you just benchmarked against only single Redis instance and claimed performance win?
Signle node of Redis vs. Single node of HPKV. so it's an apples to apples comparison
>Even if so, how do benchmarks compare against source-available competitor DragonflyDB?
Benchmark with DragonFly coming soon :)
sorry about lack of that information in documentation, we'll update that. for for now, the durability guarantee on Pro is 30 seconds. on Business with HA is 5 minutes.
1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers.
2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge.
3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers
the transactional guarantees we provide is via:
1- atomic individual operations with retries
2- various conflict resolution strategies (timestamp, etc.)
3- durability through controlled persistence cycles with configurable commit intervals
depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
A write operation on a SSD takes 10s of uS - without any VFS layers
1- as mentioned, we are not using any traditional filesystem and we're bypassing several VFS layers.
2- free space management is a combination of two RB trees, providing O(log n) for slice and O(log n + k) - k being the number of adjacent free spaces for merge.
3- majority of the write path employs a lock free design and where needed we're using per cpu write buffers
the transactional guarantees we provide is via:
1- atomic individual operations with retries
2- various conflict resolution strategies (timestamp, etc.)
3- durability through controlled persistence cycles with configurable commit intervals
depending on the plan, we provide persistence guarantee between 30 sec to 5 minutes
-Persistent by default without any performance penalties
-The pub/sub feature which is unique to HPKV and allows for a bi-directional websocket connection from clients to database
-Lower cost as we need less expensive infrastructure to provide the same service
-Simple API to use
However, it feels a bit weird: at this level of performance going SaaS only kinda defeats the purpose, no?