None of this really shook my faith in Backblaze being able to recover until I got to this part:
> Instead, Backblaze’s new CFO, Marc Suidan, joined from Beachbody (NYSE: BODI), a multi-level marketing company
eek
jmcguckin 2 days ago [-]
I remember when my mom used to hold Tupperware parties.
I guess they could go door-to-door selling backup or maybe hold Backblaze parties…
hiyer 2 days ago [-]
Beachbody isn't an MLM. They sell workout programs (used to be dvds but have now moved to a subscription model). From personal experience, I can say that at least one of their programs - Tony Horton's P90X - is very good.
lolinder 2 days ago [-]
They have been historically [0], it looks like they only changed their model last October [1], a few months after Marc Suidan switched to Backblaze.
TFA even specifies this:
> Instead they hired Marc Suidan who joined from Beachbody (NYSE: BODI) - a publicly traded health and fitness company where he was CFO between May 2022 and August 2024 when the company was operating as a “multi-level marketing” company.
My roommate freshman year of college was hocking P90X boxes and trying to get a "downstream" going. Quacks like a duck to me.
slenk 2 days ago [-]
They WERE an MLM until they stopped being able to make money. There are good YouTubers out there dedicating time to getting people out of MLMs (some are basically cults). 99% of people in MLMs lose money.
If you think he is a good CFO I think you drank the coolaid.
The workouts, including P90X, were great. The MLM part, which you and I may not have been aware of, was on the coaching side. Coaches would try to recruit other coaches to be downstream of them, and so on. Part of it was also selling their over priced protein shakes. But yes, if you were just a person who bought a P90X dvd and did the workouts, your were oblivious of this and just got in good shape and loved the products.
2 days ago [-]
cyanydeez 2 days ago [-]
MLMs focus on recruitment over product selling is what makes them an MLM. The actual product having some use is coincidence.
sixhobbits 2 days ago [-]
I know nothing about them so not saying you're wrong but the statement "X is not an MLM" is normally a positive indication that X is an MLM.
And then promoting them also gives strong mlm vibes.
So I would bet more money on them being an mlm after reading your comment than before
ellen364 2 days ago [-]
Under that rule of thumb, anything accused of being an MLM will seem like an MLM, even if the accusation is absurd.
"Uh, what?? Air isn't an MLM, it's just the thing we breath. We all need it to live."
"Huh, I guess air is an MLM after all. Wild."
hinkley 2 days ago [-]
Conversely, “they’re not an MLM, they sell a product.” Is a bullshit, non-defense statement. MLMs always have a product. That’s what keeps them from being Ponzi schemes.
derefr 2 days ago [-]
A finer distinction might be between MLMs (where the multi-level part is very literal — everyone is being garnished by their upline and profiting from their downline), vs. flat marketing organizations that just give (centralized, corporate, one-time) recruitment bonuses to their salespeople.
Vector Marketing, for example — the company that sells Cutco knives door-to-door — might be incredibly scummy, sure. Their entire business model is to
1. talk college students into thinking they can make a continuous monthly profit by selling $800 knife sets (when really the profit is one-time at best, by tapping into each college student's family and friends — who are only sympathetic enough to buy the knives [if they even are], because it's their family/friend asking);
2. forcing those college students to "buy into" the company, purchasing a knife-set of their own to use in sales demos (which they can't return if they quit);
3. and also forcing those college students to recruit other college students on campus.
But, because every transaction is ultimately just lining Vector Marketing's pockets directly — without any revenue structure involving making more money as you recruit others whose sales "become your sales" — it's not Multi Level Marketing.
It's just sleazy.
(And yes, this is just me taking this opportunity to rant about Vector Marketing. I have half a Cutco knife set laying around, from when my college roommate attended a "job offer" and got essentially bullied into buying a set before they could leave. Every time I use it, I think of how tarnished the Cutco brand is by these awful tactics of Vector Marketing's practices. Which is a shame, because they're decent knives. But obviously not knives I'd ever recommend anyone buy, because I don't want to support a company that thinks it's a good idea to work with a company like Vector Marketing.)
slenk 2 days ago [-]
BODI was an MLM that had to stop being an MLM and are hemorrhaging money
sjml 2 days ago [-]
Article says they're losing customers to Wasabi, but as far as I can see on a quick perusal of their website, Wasabi is just straight storage? For me, a lot of the benefit of Backblaze comes from a (mostly) good client solution that handles the automated backup. The backup history upsell is also great for someone (like me) whose travel patterns often mean going months without reliable internet.
I know I could use some open source stuff to get similar functionality, but I feel almost as nervous rolling my own backup as I would rolling my own crypto. Does Wasabi have some client solution that I'm missing?
esperent 2 days ago [-]
By coincidence I just spent several hours today comparing exactly these two services (Wasabi vs B2) and even though I am familiar with Backblaze and have used it before, I went with Wasabi. It's basically the same price, has more regions, a very simple interface, and rather than having to manage both hot and cold storage Wasabi is cheap enough to just throw everything into hot.
The only downside I found is that they limit free egress to 100% of your storage (so if you store 1tb, you get 1tb egress). In practice I don't think this will be an issue for my use case.
So I'm an example of a person who this morning had almost never heard of Wasabi, knows Backlaze well, and after an hour or two of research I have completely switched over.
mdasen 2 days ago [-]
Isn't B2 cheaper than Wasabi? B2 is $6/TB while Wasabi is $6.99/TB. Why would you have to manage hot and cold storage with B2?
Wasabi also treats every object as having a minimum 90-day storage period. So if you upload something and delete it, it's still considered "stored" for billing purposes for that 90-day minimum.
B2 also gives you 3x storage as free transfer compared to 1x for Wasabi.
I'm not saying you shouldn't go with Wasabi, but the reasoning you've articulated doesn't seem to track.
masfuerte 2 days ago [-]
Additionally, Wasabi bills you for a minimum of 1TB of active storage, which is $6.99/month. For my modest backup needs it is way cheaper to use any service that bills per GB.
Wasabi also charges objects at a minimum 4KB of storage, which isn’t great if you have millions of files at 1-2KB.
gruez 2 days ago [-]
>Article says they're losing customers to Wasabi, but as far as I can see on a quick perusal of their website, Wasabi is just straight storage?
That's addressed in the article:
>Since 2021, Backblaze’s B2 Cloud Storage segment revenue growth has outpaced its legacy Computer Backup segment. In Q4 2024, the company announced that its B2 Cloud Storage segment generated $17.1 million in revenue, surpassing its Computer Backup revenue of $16.7 million.
Even if Wasabi isn't a straight replacement for Backblaze's entire business, it's a replacement for its biggest and fastest growing segment.
nikisweeting 2 days ago [-]
how in the world do they only make $17.1 in revenue, everyone I know is on BackBlaze, even if they're only used by techy people I'm surprised they're not making $100m+/quarter.
gruez 2 days ago [-]
Anecdotally most people, even "techy people" don't do actual backups. At best, they have icloud/google backup turned on for their phone, and upload important files are on dropbox/onedrive.
nikisweeting 1 days ago [-]
I would've guessed their B2 business is much bigger than the personal backups side.
geerlingguy 2 days ago [-]
They're cheap (the reason many people use them), and the number of people we know that use them is ridiculously outsized compared to the general population, because we know people in the tech sphere.
conradfr 2 days ago [-]
Maybe you know their entire customers base.
I know nobody who use them, so it all balances itself.
anon7000 2 days ago [-]
I mean, my B2 monthly bill is like 17 cents.
If they don’t have enterprises and people storing large amounts of data...
ramraj07 2 days ago [-]
Turns out big enterprise was right in not adopting such a small player lol.
j45 2 days ago [-]
Appreciate the intro to Wasabi, happy to learn about others too while evaluating backblaze.
e40 1 days ago [-]
I used to be a customer of computer backup, but had issues with it and their support suggested I delete my backup and start over with a new account. I was flabbergasted. On macOS btw.
I ended up rolling my own solution with rclone and testing b2 and r3, going with b2 because of the price and speed.
sandbags 1 days ago [-]
Years ago I used Backblaze and it twice "lost" my entire backup with their support staff shrugging and suggesting I start it again. At that point I didn't have fibre so a full backup was weeks if not months. Needly to say I opted out of their service. I am not overly surprised that this is not a solved problem.
e40 21 hours ago [-]
Yeah, it's clear it has bugs and doesn't work well. rclone to B2 works fine for me. And I tested with Cloudflare R2 and can switch to it if BB goes under.
R2 was slower than B2, in my tests. I think it's slightly more expensive, too, but not much IIRC.
tw04 2 days ago [-]
You’re missing the point. Enterprise customers where they would presumably see the bulk of their profits is what they’re losing to wasabi. Commercial “unlimited backups” is not a wildly profitable business which is why basically every other competitor in the space either failed or intentionally exited the market segment.
sjml 2 days ago [-]
Ah, got it; that makes sense. Thanks for the clarification.
dehrmann 2 days ago [-]
> but I feel almost as nervous rolling my own backup as I would rolling my own crypto
Is there a Restic frontend for Windows that you'd be comfortable setting up a family member with?
Backblaze business is 50% object storage (from financial statements)
That's where Wasabi is taking customers
lud_lite 2 days ago [-]
As a customer do I need to worry? It is my favourite cheapo backup.
klodolph 2 days ago [-]
I recommend worrying about any service where you don’t pay a fee that scales with usage. This includes Backblaze. Yes, I recommend worrying about Backblaze and I’ve recommended worrying about it for a while.
Storage costs money. People love to dream up creative business models where your personal storage is subsidized by some other part of the business, but I think it’s just a matter of time before the business model changes. At the end of the day, there’s a massive gravitational pull that brings everything in line with market rates. These days, I think we have a pretty good idea of what market rates are for cloud storage. Anything less than market rates should be viewed with suspicion.
From another angle, in any long-term relationship with a vendor, you want the vendor to make money. We know that major cloud vendors do, or at least close enough, because they charge the same for small customers and big ones (or close enough). None of them offer unlimited data… or at least, not any more (most of them used to, but those parts of the business always get shut down).
rsync 2 days ago [-]
"I recommend worrying about any service where you don’t pay a fee that scales with usage."
This is the issue and it is a heuristic that people need to develop.
Your provider especially your backups provider needs to have interests that are aligned with your interests.
The non-B2 portion of Backblaze has a financial incentive to keep you from using their product and to use as little of their space as possible. This is a bad, misaligned set of incentives.
On the B2 side, all of our suspicions about the (well timed) "me too" IPO have been confirmed with each and every quarterly report showing increased debt load and the classic "losing money but making up for it on volume".
Now that money isn't free anymore it's just a matter of time.
The sad part is they don't use standard, commodity JBOD enclosures so there won't even be anything useful in the fire sale :(
klodolph 2 days ago [-]
> Your provider especially your backups provider needs to have interests that are aligned with your interests.
I also think about all those cafés with free WiFi, where you buy a single coffee and take up a table for six hours while you browse Facebook and have your screenplay open in the background. I benefit, but the café doesn’t.
justinclift 1 days ago [-]
Don't cafe's tell people to move on if they up space for large amounts of time when the place is busy?
mindslight 22 hours ago [-]
> The sad part is they don't use standard, commodity JBOD enclosures so there won't even be anything useful in the fire sale :(
Can you elaborate on this? They mount in a standard rack and it looks like they have a backplane connected to a commodity motherboard. I'd think if you wanted it to be a JBOD you could replace the motherboard with an external SAS connector. Based on your nick I'm sure you've thought this through so I'm curious.
The only thing I see is that it looks like you have to slide the whole unit out to swap a drive, but that seems like an inconvenience rather than a show stopper.
gruez 2 days ago [-]
>These days, I think we have a pretty good idea of what market rates are for cloud storage. Anything less than market rates should be viewed with suspicion.
Are you talking about their B2 product or their backup SaaS? The former has "fee that scales with usage", and the latter probably has enough normal users (ie. not data hoarders backing up 50 TB on the $99/year plan that they're not losing money overall.
$99/year with no limits on data, as far as I can tell. The context here is someone talking about personal backups, not Backblaze’s broader offerings. We know the market rates for cloud storage, more or less, since they’ve converged somewhat, across the industry. This makes it easy to calculate what kind of usage patterns would lead someone to save money at Backblaze.
Cloud services often have a free tier or cheap tier to attract new customers. IMO, this is not it. This is a product. These people are not turning around and signing contracts with Backblaze at work. At least, not many.
So it could be that people are just using it below its limits, or it could be that it’s subsidized by other parts of the business. Overall P&L is irrelevant—you want to be able to explain this product as a profitable product, as some viable sales channel, or as marketing.
My recommendation is to buy storage products where the company is making money off you, or nearly.
ricardobeat 2 days ago [-]
The backup plan has no API access (you need to use B2 which is usage-billed), it’s for a single machine, and external drive data is lost after 30 days from last scan which makes it very inconvenient for anything other than.. backing up a computer’s hard drive. Nothing indicates misuse of this plan is causing elevated operating costs for them.
klodolph 2 days ago [-]
I’m aware. That isn’t relevant to my analysis.
immibis 2 days ago [-]
I believe they're public about the fact they have some users who back up hundreds of TBs and they're still willing to just deal with it to keep the flat price structure.
gruez 2 days ago [-]
>$99/year with no limits on data, as far as I can tell. The context here is someone talking about personal backups, not Backblaze’s broader offerings. We know the market rates for cloud storage, more or less, since they’ve converged somewhat, across the industry. This makes it easy to calculate what kind of usage patterns would lead someone to save money at Backblaze.
So do the calculations? As I mentioned in another comment, their unit economics are sound, with them making more than 50% gross margin.
>So it could be that people are just using it below its limits
That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
klodolph 2 days ago [-]
> So do the calculations?
Is this some kind of personal challenge? It’s arithmetic for chrissakes, not calculus. S3 IA $0.0125/Gb-mo, ingress basically free, $99/yr÷(12mo/yr×$0.0125/Gb-mo)=660 Gb. Reasonable to ignore egress for backups—you may want to account for it, but it’s reasonable to ignore it. Do your own calculations if you have different assumptions.
That’s not hard. The numbers don’t have to be exact.
Maybe I just think it’s easy because I’ve run numbers like these for a living. Given X different storage configurations, under which conditions is configuration 1 cheaper than configurations 2 and 3?
> That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
The economics of gyms are different because the cost is not only the membership, but also the time it takes you to get to the gym. Cloud services are a much more efficient market, which means the margins are much lower.
gruez 2 days ago [-]
>Is this some kind of personal challenge? It’s arithmetic for chrissakes, not calculus. S3 IA $0.0125/Gb-mo, ingress basically free, $99/yr÷(12mo/yr×$0.0125/Gb-mo)=660 Gb.
This includes AWS's margin, which could be quite handsome. Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Even taking the 660GB figure at face value, that's plausibly within the range of what I'd expect the average person to have backed up, especially when you consider that the pricing is per machine, and the standard desktop/laptop comes with around 500 GB of storage.
You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
>The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
It's a backup service. It's not even trying to compare itself with the likes of dropbox or onedrive. You should have at the very least a second copy locally.
klodolph 1 days ago [-]
> This includes AWS's margin, which could be quite handsome.
It could not be.
Storage costs are more or less race to the bottom. Storage business model is that it’s cheap and commodified, but sticky. Your data is stuck in S3 because it’s a pain in the ass to migrate. The margins come from non-commodified parts of the cloud (higher-level SaaS products) and I suspect egress costs (I don’t have the details).
There’s three parts to my understanding here. One part is my understanding of the business model, which explains why storage is a race to the bottom, why it’s not subsidized (or not subsidized much), why from a strategic perspective, the margins are getting eliminated. The second part to my understanding is the technical aspect—the calculations for how much it costs to stick hard drives in a data center and run storage systems on top with the desired accessibility and reliability, and I have some fairly deep expertise in parts of this. The last part is my personal experience working for cloud providers.
I can elaborate on the first two parts but for obvious reasons some of my knowledge is proprietary and I can’t share it.
> Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Bandwidth is not commodified like storage. I know that storage cost is close to operational cost because I’ve done extensive calculations and comparisons—and some projects are cheap enough that you have to revisit your theories about how they work on a technical level.
I don’t know why bandwidth is expensive. I’m not an expert on bandwidth or CPU or 90% of the other stuff. I only happen to have expertise on storage.
There are services you can buy where egress is the major cost and egress is significantly cheaper than egress from AWS, GCP, or Azure. Either egress is a complicated product or the margins are high. Probably some combination. Storage is less complicated and the margins are lower.
> You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
I guess I don’t understand what point you’re making, or how that point about gross margins makes sense. It doesn’t make sense to me so I’m not sure what I’m supposed to respond to.
bobmcnamara 1 days ago [-]
> There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly.
There's no way that planet fitness can equip and operate a gym for $10/month per member, if one of them simultaneously used all machines at all locations.
mingus88 2 days ago [-]
According to the article they _are_ losing money overall, every quarter since the IPo
gruez 2 days ago [-]
I meant the unit economics of their backup product, not the company as a whole. They made $69M in gross profit in 2024, with a gross margin of 55%. That turns into a lost only after subtracting R&D ($42M), sales and marketing ($44M), and G&A ($29M). Of course, those expenses can't be ignored in the context of a company that's set out to make money, but at least they're not selling $10 worth of storage for $5, like the OP implied.
Spooky23 2 days ago [-]
I’d make the exception for Google Drive, OneDrive and whatever the AWS one is. The hyperscalers are able to get prices way cheaper with economies of scale and price models in a sustainable way.
When O365 launched, they were using spinning disk for exchange. The issue was that they stranded capacity because of the IOPS needs of exchange. So “free”, (low iops) SharePoint and OneDrive for business data utilized that “free” capacity.
rsync 2 days ago [-]
"I’d make the exception for Google Drive, OneDrive and whatever the AWS one is."
No - no exceptions.
We must, as sophisticated 21st century actors, insist that providers (especially providers of critical services) have financial interests that align with our own interests.
I don't care how big the parent is or how much money they have to burn - if it makes financial sense to keep you from storing your backups you need to go elsewhere.
Spooky23 2 days ago [-]
You’re missing the point. Those companies are providing extremely profitable services, of which data storage is a part. Office and Workspace are lines of business with 50-60% margins.
Non-nerds are going to screw up backups without good UI. If you want to pay by the drink, like I said, use the hyperscalers.
immibis 2 days ago [-]
On the other hand why should you not extract value from someone who is giving you free value?
Spooky23 2 days ago [-]
Unless you uuencode your photos and chisel them into stone tablets, stored in your personal salt mine, you’re not serious about backups.
j45 2 days ago [-]
The economies of scale go into profit for them while still incrementally increasing costs for clients.
If it was aligned with economies of scale, customers would get more storage for the same price every year.
We don't hear about cloud prices coming down that often.
lud_lite 2 days ago [-]
Thanks! I suspect that for my use case (actual backup no meed for cloud access) s3 glacier is statistically cheaper.
gruez 2 days ago [-]
>actual backup no meed for cloud access
How do you test your backups? You do test your backups, right? Your other comment mentions using dropbox, but that's hardly a real backup solution, and you could easily run into a situation where you need to retrieve files beyond the 30 day window.
mingus88 2 days ago [-]
If you regularly test your backups, glacier gets a lot more expensive due to egress fees
And it’s such a good thing to remind people of. You don’t manage a backup service. Nobody cares about backups.
People care about restores.
klodolph 2 days ago [-]
Glacier egress is less than it used to be, and you can use a combination of glacier + other storage classes using lifecycle rules.
I would personally be fine running tests on data that is recently backed up and only testing the data in glacier once or twice. Think about why you test backups in the first place—the main errors you’re trying to catch are problems like misconfiguration, backups not getting scheduled or not running, or not having access to your encryption keys. You can put your most recent backup in “infrequent access” and let older objects age out to glacier with lifecycle rules.
Glacier used to have really expensive retrieval costs. That’s now called “glacier deep archive” and as far as I know, major use cases are things like corporate recordkeeping / compliance (e.g. Sarbanes-Oxley). The costumers for deep archive should be sophisticated customers.
leidenfrost 2 days ago [-]
What do you recommend as an alternative? I use it as a cheaper replacement to S3
geerlingguy 2 days ago [-]
I backup to Amazon Glacier S3 Deep Archive backed buckets. The price is only a little higher for a few TB than other providers, though egress fees in case of restore are a lot higher. That's a tradeoff I'm willing to make as I'd have two have two separate local NAS replicas corrupt or die before I need to rely on the Deep Archive.
Spooky23 2 days ago [-]
For raw backup, I use GCP. Pricing is similar to AWS and the deep storage is easier to use.
For most regular people, I suggest Google Drive or OneDrive because you get the value add of the their ecosystem. With Google, Photos are good. With Microsoft, the Office+OneDrive subscription is a great value.
billfor 2 days ago [-]
IDrive.
j45 2 days ago [-]
Storage is a fixed fee (space on a hard drive). It make money every month when it sits there and is used.
Bandwidth costs can seem like a lot more, but when you purchase a 10 gig fibre, you have unlimited data, up until the full speed of the 10 gig fibre, 24/7. That number in TB can be calculated.
Clouds massively mark up services. Ones that underprice can have the opposite problem.
I have heard good things about the Backblaze service itself, and appreciate the hard drive reviews they put out.
immibis 2 days ago [-]
You should worry they'll increase the price and worry they'll delete your data if you don't pay the higher price. It's not necessary to worry they'll go out of business just because they are charging a fixed price. They'll raise the price long before going out of business.
j45 2 days ago [-]
Having your own NAS in the mix more and more is unavoidable.
On one side, the cloud is someone else's computer they will always charge more an more for it because customers have not learned the economics of cloud hosting and how profitable it is on the other end.
On the other side, a NAS is your storage computer, in a simplified home appliance form. Connect it to back up to any cloud you wish.
They're not at risk of going bankrupt. This is more about confidence in the company to go anywhere that would help the share price.
The company is being plundered and run for the benefit of the exec team printing shares for themselves. Nobody should be buying shares expecting the price to rise.
stego-tech 2 days ago [-]
If the service isn’t making money consistently, I would worry about its long-term viability. This is basic commoditized storage they’re managing to screw up making a profit on, not some super-niche or brand-new product line.
It’s bits in a barn, and they’re not making rent. That’s worried me enough to re-evaluate my relationship with them, especially in light of this research.
PaywallBuster 1 days ago [-]
They already increased price from $5 to $6 sometime ago
Financial statements indicate they selling services at 50% margin, but all the extra sales/admin/R&D add up to a loss
They could cut all of that I guess, it's not like object storage need a lot of innovation and they've already stopped making their own storage servers and
dist-epoch 2 days ago [-]
If it's your only backup, yes, you definitely should.
lud_lite 2 days ago [-]
It's one of two backups. The software is great. Dropbox level of just works. iDrive by comparison is cheaper but always errors and files it couldn't touch.
nikisweeting 2 days ago [-]
Yeah do NOT do iDrive if you use git or care about symlinks, I learned the hard way. Backblaze doesn't backup symlinks either for that matter...
rsync 2 days ago [-]
"Yeah do NOT do iDrive if you use git or care about symlinks, I learned the hard way. Backblaze doesn't backup symlinks either for that matter..."
Gosh, if only there were a cloud storage provider that gave you an empty UNIX filesystem you could do anything you wanted with ... maybe something built totally on open standards that could handle hardlinks and symlinks and such ?
If only ...
nikisweeting 1 days ago [-]
I've wanted to use ya'll for a long time but the ~2x price compared to B2 has always stopped me. A remote ZFS cloud really is such an attractive idea though... maybe it's time to reconsider after this article.
cantrecallmypwd 2 days ago [-]
tarsnap, mega ;)
rsync 2 days ago [-]
The world would be a better place if more people used tarsnap…
lud_lite 2 days ago [-]
It is expensive. If we do a monthly test restore like someone suggested then 1TB costs $6000/y. No restores is $3000/y. s3 regular is 10% of that.
mingus88 2 days ago [-]
I use restic with b2 as a backend
It’s not the same service as backblaze’s client but it does everything I need, with dedupe
nikisweeting 1 days ago [-]
Yeah I use duplicati, basically the same, it slow but works well and has saved my bacon a few times in real-world situations.
mikeryan 2 days ago [-]
Totally an aside but is Morpheus what happened to the Hindenburg Research guys after the founder of that stepped away? The site is eerily similar and the timing makes sense. I loved reading Hindenburg’s research.
Funny, I wrote this without seeing what the full URL was (I’m on my phone) so the use of the word “gracious” preceded me knowing the path was /gratitude
SSLy 1 days ago [-]
To me, the Hindenburg Research announcement seemed to suggest that they were being threatened by the subjects of their research.
guerrilla 2 days ago [-]
Well if they go out of business where the hell am I going to get my hard drive statistics from? ;D
tgtweak 2 days ago [-]
Did they cook the books on those too?
honestly I participated in the IPO then sold when it was up a bit - I had faith in their fundamentals as a business but they did not seem mature as a public company.
guerrilla 2 days ago [-]
I don't know what the incentive would be for doing that. Could they use it to drive prices on HDDs lower somehow? I vaguely recall them being activist as far as quality.
rdtsc 2 days ago [-]
> While Budman claimed that he wanted to avoid any appearance of trying to “time the market,” he appears to have done so nearly perfectly, as Backblaze’s shares crashed by as much as 26% intraday following the announcement and have continued to slide ever since.[5]
Doesn’t that seem like insider trading? I guess he can claim he didn’t know so and so was going to quit.
hinkley 2 days ago [-]
Not to say he’s innocent, but insiders can also trigger market crashes just by selling their stock. There are public records of substantial holdings by company insiders and while I don’t think they’re made available immediately, they are updated periodically and people can and do read into it.
That’s part of why they usually deputize a brokerage to periodically sell a fraction of their holdings at intervals.
It’s been a long while since I paid any attention to this but I do believe I saw at least a couple of instances of an insider triggering weakness in their stock just by increasing the rate of sale of their holdings.
rdtsc 2 days ago [-]
Yeah it would have to be something like that. It seems a too obvious of a move, so I figured he might have had a solid loophole.
hinkley 2 days ago [-]
To wit: Bill Gates did a bunch of press before starting his foundation to make it clear that him selling MS shares was not about concerns over MS’s future but that he needed a shitload of money for other pursuits.
And Warren Buffet would use many frontmen to slowly buy shares without triggering a buying frenzy. You can’t just buy or sell 3% of a company without causing drama.
Spooky23 2 days ago [-]
It almost certainly is. Canceling the planned sale makes it look even worse.
But… are there securities laws in this era where policy decisions are timed to pump/dump the lowest?
PeterStuer 2 days ago [-]
Wasabi a more known brand than Backblaze in the storage space? Doubt!
mdasen 2 days ago [-]
It might depend on who you are. I think Backblaze is well known on here because they've been around for longer and their hard drive stats posts are well liked.
If you're a less technical CTO, Wasabi seems to do a lot more traditional marketing on things like sports games: https://ibb.co/21qNkM9b (NBA, NHL, Premier League soccer). I'm not saying that makes them better, but a lot of people with purchasing power are going to be seeing Wasabi around.
Wasabi has also been pushing value-add services on top of their storage with Wasabi AiR (https://wasabi.com/cloud-object-storage/wasabi-air). It's basically an AI-powered database of your content. If you're running a place generating lots of content, having a system automatically tag and make searchable all those images and videos is a really nice value-add. Being able to search for a video clip containing a specific person in your library junk pile of video can be nicely useful for a lot of organizations.
It could just be that different people get their brand awareness from different places.
avsteele 2 days ago [-]
Very happy with their B2 storage. Easy to use, works well, prices fine. I use it as a backup target for several of my Synology NAS's. I hope they stay in business
Fizzadar 2 days ago [-]
Moved to wasabi from their s3-like b2 product because it was so utterly unreliable. Pretty wild list of things if true.
e40 1 days ago [-]
I use b2 and never had any issues. What problems did you have?
everfrustrated 2 days ago [-]
Morpheus Research should look at Fastly next.
sndean 2 days ago [-]
Can anyone recommend a decent Backblaze alternative in case things really go south? Not so much for storage but for the automated computer backups that I have Backblaze doing. I'm okay paying more than Backblaze's $9/month.
wmf 2 days ago [-]
I haven't tried it but Arq has a good reputation.
tinodb 1 days ago [-]
I’m using Arq which backs up to a Hetzner storage box. Cheapest €/TB I could find.
ryao 2 days ago [-]
I have no experience with them but I have heard good things about rsync.net. They use ZFS.
cantrecallmypwd 2 days ago [-]
Much data, medium risk tolerance: Mega
Small data, or low risk tolerance: Tarsnap
rsyring 23 hours ago [-]
Tarsnap restore times are crazy slow. Avoid.
We replaced with Borg.
bartvbl 2 days ago [-]
I use Jottacloud, which is quite similar in many respects.
memset 2 days ago [-]
Who are the players in the “s3 compatible but cheaper than AWS” space?
- Wasabi
- idrive
Who else?
manquer 2 days ago [-]
Almost every cloud provider outside of Azure and GCP have an s3 compatible offering . S3 API is de-facto standard for object storage tooling
Cloudflare, scaleway, hetzner, OVH, Oracle, minio and dozens of others all have s3 compatible object stores
Most of them expect AWS have at least inter cloud transfer free (or discount Azure and GCP) being part of bandwidth alliance(BWA).
AWS S3 pricing is designed to keep you in the ecosystem , not be the cheapest, so tend to be costliest not cheapest unless your workload is archival.
For almost all non archival workloads egress costs will be on par or much higher than storage costs and AWS[1] charges a lot for any bandwidth.
AWS also makes migration really expensive so if you started with them, you are likely stuck as you would have to pay up to 10-12 months of cost to migrate out if you are on anything but the standard tier (cheaper the tiers higher the retrieval costs)
[1] All tier one providers have 10-20x b/w costs compared to tier two , however Azure and GCP discount for inter cloud transfers AWS does not
From the article (TLDR: the author has a financial interest in seeing their stock decline)
> Initial Disclosure: After extensive research, we believe the evidence justifies a short position in shares of Backblaze (NASDAQ: BLZE). Morpheus Research holds short positions in BLZE, and Morpheus Research may profit from short positions held by others. This report represents our opinion, and we encourage all readers to do their own due diligence. Please see our full disclaimer at the bottom of the report.
Spooky23 2 days ago [-]
Sure, but I’d more worried that the founders and management seem like they are short. The CEO left, announced a big stock sale, cancelled it, then dumped a bunch coincidentally when the CFO left (but before disclosure). Then the company hired an MLM CFO.
Oh, and the finance employees refused to sign off on the books and a few are suing the company.
It sucks, but Backblaze is cooked.
cj 2 days ago [-]
Sure, I agree.
But this is a hit piece. For any hit piece, I think it’s important for readers to know the motivations of the author, regardless of the validity of the claims.
gruez 2 days ago [-]
That's basically how every short report works. Finding corporate malfeasance and writing a report costs money, and Backblaze isn't famous enough that journalists from the new york times or propublica would go after them. So the only people with incentive to expose such corporate malfeasance are short sellers.
dist-epoch 2 days ago [-]
And contrary to popular opinion, Taleb thinks that you should put more weight on such opinions, because they have skin in the game and are willing to back their beliefs with money. Short selling is not risk free, if they are wrong, they will get burned.
hobs 2 days ago [-]
I mean, just read the article - if half the stuff stated is true its pretty bad, though its interesting how much of the drama is centered around 2022.
tw04 2 days ago [-]
If half the stuff stated is not true, he’ll assuredly be facing a lawsuit.
jillyboel 2 days ago [-]
And writing articles like this makes it more likely they become right
Spooky23 2 days ago [-]
Well yeah, when people find out that someone is stealing from them, they tend to not want to do business.
3eb7988a1663 2 days ago [-]
That is how short selling works. Do lots of research, find some fatal flaw in a business model that everyone else has missed, and put money on the position. The final step is you have to advertise your findings, otherwise you are sitting on secret knowledge which the market will not price appropriately.
dmtroyer 2 days ago [-]
I would have appreciated seeing this at the top of the article.
jmcguckin 2 days ago [-]
I think Backblazes pricing model is unsustainable and expect them (and any competitors who mimic their model) to eventually fail when the money runs out.
Racks and racks of disk drives use a lot of power and cost a lot to keep running. A better model would be storage based on tape; one of those large tape libraries (e.g. iceberg) with nearline disk space. Most of the data that gets uploaded never gets accessed, so keeping it on tape instead of spinning rust makes sense. Sure, there’s tradeoffs, but imagine how much they’d save if they could downsize their data center spending by 90%.
Havoc 2 days ago [-]
sigh...literally just recommended it to a family member as the simple option
EasyMark 2 days ago [-]
do some research, this article smells of hit piece, especially since the author is set to make money off of bad publicity for the company.
nodesocket 2 days ago [-]
You should not stop recommending Backblaze due to a short seller's opinion piece. What are their motivations?
aghilmort 2 days ago [-]
ugh backblaze was great alt to S3 the heroku of storage for a while, used them often few years ago
erichocean 2 days ago [-]
Short-seller writes disparaging article to move market, news at 11.
(At least they disclose it at the bottom, but it should be in the title.)
PaywallBuster 2 days ago [-]
tl;dr
this is activist short seller report/investigation, some of the points made:
- never been profitable
- execs selling aggressively
- loosing customers to Wasabi
- accused of cooking the books
- being sued by former execs for anti whistleblower/wrongful termination
- execs leaving
ryao 2 days ago [-]
Wasabi uses ZFS while backblaze does not. I wonder if this in any way contributed to the cost differences.
Twirrim 2 days ago [-]
Backblaze uses erasure encoding, which is currently the best and most efficient way to do storage. It's how every major object storage platform works.
The very quick high level explanation is that in storage you talk about "stretch factor". For every byte of file, how many bytes do you have to store to get the desired durability. If your approach to durability is you make 3 copies, that's a 3x stretch factor. Assuming you're smart, you'll have these spread across different servers, or at least different hard disks, so you'd be able to tolerate the loss of 2 servers.
With erasure encoding you apply a mathematical transformation to the incoming object and shard it up. Out of those shards you need to retrieve a certain number to be able to reproduce the original object. The number of shards you produce and how many you need to recreate the original are configurable. Let's say it shards to 12, and you need 9 to recreate. The amount of storage that takes up is the ratio 9:12, so that's a 1.3x. For every byte that comes in, you need to store just 1.3x bytes.
As before you'd scatter them across 12 shards and only needing any 9 means you can tolerate losing 3 hard disks (servers?) and still be able to retrieve the original object. That's better durability despite taking up 2.7x less storage.
The drawback is that to retrieve the object, you have to fetch shards from 9 different locations and apply the transformation to recreate the original object, which adds a small bit of latency, but it's largely negligible these days. The cost of extra servers for your retrieval layer is significantly less than a storage server, and you wouldn't need anywhere near the same number as you'd otherwise need for storage.
The underlying file system doesn't really have any appreciable impact under those circumstances. I'd argue ZFS is probably even worse, because you're spending more resources on overhead. You want something as fast and lightweight as possible. Your fixity checks will catch any degradation in shards, and recreating shards in the case of failure is pretty cheap.
srean 2 days ago [-]
> It's how every major object storage platform works.
Very interesting. Could you name a few, am curious. I would be happy if erasure codes are actually being used commercially.
What I find interesting is the interaction of compression and durability -- if you lose a few compressed bytes to reconstruction error, you lose a little more than a few. Seems right up rate-distortion alley.
Twirrim 2 days ago [-]
That I know of (and is public so I'm not breaching any NDA) AWS S3[1], Azure[2], GCP[3], Backblaze[4], Facebook's storage layer uses it[5][6], and Oracle Cloud's Object Storage Platform[7].
The economies of scale mean that you really have to have something like erasure encoding in place to operate at large scale. The biggest single cost for cloud providers is the per-rack operational costs, so keeping the number of racks down is critical.
Ceph has a very stable EC feature. And lot of companies use Ceph as a storage backend. Unfortunately I cannot find any straightforward statement about a commercial offering, but I would bet that DreamHost's DreamObjects does use it.
While it's not "commercial", but CERN uses it and many other institutions.
XFS or ext2 (or 3 or 4 wo journal), without LVM or mdraid.
There's no point to adding RAID at the hardware or OS level for object storage boxes when redundancy exists at the application level. A drive with too many errors will be marked "dead" and just spun down and ignored.
Metadata servers OTOH tend to be engineered to be much more reliable beasts.
ksec 2 days ago [-]
>Wasabi uses ZFS
Can't seems to find anything specific about Wasabi uses ZFS on Google. And Wasabi doesn't change you on egress. So I guess they are similar in terms of pricing.
Although B2 seems to be way more popular on HN. I rarely see Wasabi here.
ryao 2 days ago [-]
There is the testimonial on Klara Systems’ website:
"The developers at Klara Inc. have been outstanding in helping Wasabi resolve ZFS issues and improve performance. Their understanding of ZFS internals is unmatched anywhere in the industry" - Jeff Flowers, CTO, Wasabi Technologies
You could also search the OpenZFS repository for commits with the word Wasabi in them.
badlibrarian 2 days ago [-]
Book price is $6.99/TB for Wasabi vs $6/TB for Backblaze. Wasabi charges 90 days minimum for storage, and egress bandwidth is limited (honor system) to your total monthly data storage amount.
thayne 2 days ago [-]
Wasabi also requires you to pay for a minimum of 1TB, whereas B2 charges per GB. That doesn't really matter for a company using a ton of storage, but it does for my personal use case of a few tens of GB.
Backblaze don’t charge egress if you put it behind Cloudflare (even the free tier).
alabastervlog 2 days ago [-]
You won’t, however, be able to serve most e.g. media files or binaries this way, nor serve to clients like mobile apps, while staying within the bounds of Cloudflare’s terms for their free and “self-serve” tier paid accounts.
(Unless something’s changed since the last time I checked)
Sytten 2 days ago [-]
I dont think that would be right otherwise they R2 offering would be kinda useless.
I feel the restriction was on video/streaming.
EDIT: OP is correct for CDN but if you use R2 even as a transparent copy from another S3 like provider it is allowed [1]
I suspect it does. When I was evaluating Wasabi years ago, the sales engineers were very interested in knowing what specific kind of data we had and how compressible it was. So my guess (pure speculation) at the time was they use ZFS compression internally but charge customers for the uncompressed size.
Twirrim 2 days ago [-]
If they care about compression at the ZFS level, that means your file is going to be visible to anyone able to log in to the server, because they're relying on (at best) encryption at rest. That's not a great security model for a storage service. You don't want anyone to be able to log in to a server and see your actual files unencrypted.
If they're going to compress/decompress, ideally you'd want them to have that at the point of ingestion, then encrypted, and then store that on the target drive.
That way you can put very strong controls (audit, architecture, software etc) around your encryption and decryption, and be at reduced risk from someone getting access to storage servers.
dist-epoch 2 days ago [-]
most of the files that matter, the big ones, are already compressed/high entropy - images, videos, ...
k8sToGo 2 days ago [-]
Wasabi also has a minimum storage requirement of 90 days
mkj 2 days ago [-]
Wasabi doesn't seem to be a public company, so are they profitable?
Rendered at 14:57:21 GMT+0000 (Coordinated Universal Time) with Vercel.
> Instead, Backblaze’s new CFO, Marc Suidan, joined from Beachbody (NYSE: BODI), a multi-level marketing company
eek
I guess they could go door-to-door selling backup or maybe hold Backblaze parties…
TFA even specifies this:
> Instead they hired Marc Suidan who joined from Beachbody (NYSE: BODI) - a publicly traded health and fitness company where he was CFO between May 2022 and August 2024 when the company was operating as a “multi-level marketing” company.
[0] https://www.cnbc.com/2011/01/31/beachbody-grows-exponentiall...
[1] https://nypost.com/2024/10/01/business/beachbody-lays-off-th...
If you think he is a good CFO I think you drank the coolaid.
https://marencrowley.com/podcast/the-demise-of-the-beachbody...
And then promoting them also gives strong mlm vibes.
So I would bet more money on them being an mlm after reading your comment than before
"Uh, what?? Air isn't an MLM, it's just the thing we breath. We all need it to live."
"Huh, I guess air is an MLM after all. Wild."
Vector Marketing, for example — the company that sells Cutco knives door-to-door — might be incredibly scummy, sure. Their entire business model is to
1. talk college students into thinking they can make a continuous monthly profit by selling $800 knife sets (when really the profit is one-time at best, by tapping into each college student's family and friends — who are only sympathetic enough to buy the knives [if they even are], because it's their family/friend asking);
2. forcing those college students to "buy into" the company, purchasing a knife-set of their own to use in sales demos (which they can't return if they quit);
3. and also forcing those college students to recruit other college students on campus.
But, because every transaction is ultimately just lining Vector Marketing's pockets directly — without any revenue structure involving making more money as you recruit others whose sales "become your sales" — it's not Multi Level Marketing.
It's just sleazy.
(And yes, this is just me taking this opportunity to rant about Vector Marketing. I have half a Cutco knife set laying around, from when my college roommate attended a "job offer" and got essentially bullied into buying a set before they could leave. Every time I use it, I think of how tarnished the Cutco brand is by these awful tactics of Vector Marketing's practices. Which is a shame, because they're decent knives. But obviously not knives I'd ever recommend anyone buy, because I don't want to support a company that thinks it's a good idea to work with a company like Vector Marketing.)
I know I could use some open source stuff to get similar functionality, but I feel almost as nervous rolling my own backup as I would rolling my own crypto. Does Wasabi have some client solution that I'm missing?
The only downside I found is that they limit free egress to 100% of your storage (so if you store 1tb, you get 1tb egress). In practice I don't think this will be an issue for my use case.
So I'm an example of a person who this morning had almost never heard of Wasabi, knows Backlaze well, and after an hour or two of research I have completely switched over.
Wasabi also treats every object as having a minimum 90-day storage period. So if you upload something and delete it, it's still considered "stored" for billing purposes for that 90-day minimum.
B2 also gives you 3x storage as free transfer compared to 1x for Wasabi.
I'm not saying you shouldn't go with Wasabi, but the reasoning you've articulated doesn't seem to track.
https://wasabi.com/pricing/faq
That's addressed in the article:
>Since 2021, Backblaze’s B2 Cloud Storage segment revenue growth has outpaced its legacy Computer Backup segment. In Q4 2024, the company announced that its B2 Cloud Storage segment generated $17.1 million in revenue, surpassing its Computer Backup revenue of $16.7 million.
Even if Wasabi isn't a straight replacement for Backblaze's entire business, it's a replacement for its biggest and fastest growing segment.
I know nobody who use them, so it all balances itself.
If they don’t have enterprises and people storing large amounts of data...
I ended up rolling my own solution with rclone and testing b2 and r3, going with b2 because of the price and speed.
R2 was slower than B2, in my tests. I think it's slightly more expensive, too, but not much IIRC.
Is there a Restic frontend for Windows that you'd be comfortable setting up a family member with?
That's where Wasabi is taking customers
Storage costs money. People love to dream up creative business models where your personal storage is subsidized by some other part of the business, but I think it’s just a matter of time before the business model changes. At the end of the day, there’s a massive gravitational pull that brings everything in line with market rates. These days, I think we have a pretty good idea of what market rates are for cloud storage. Anything less than market rates should be viewed with suspicion.
From another angle, in any long-term relationship with a vendor, you want the vendor to make money. We know that major cloud vendors do, or at least close enough, because they charge the same for small customers and big ones (or close enough). None of them offer unlimited data… or at least, not any more (most of them used to, but those parts of the business always get shut down).
This is the issue and it is a heuristic that people need to develop.
Your provider especially your backups provider needs to have interests that are aligned with your interests.
The non-B2 portion of Backblaze has a financial incentive to keep you from using their product and to use as little of their space as possible. This is a bad, misaligned set of incentives.
On the B2 side, all of our suspicions about the (well timed) "me too" IPO have been confirmed with each and every quarterly report showing increased debt load and the classic "losing money but making up for it on volume".
Now that money isn't free anymore it's just a matter of time.
The sad part is they don't use standard, commodity JBOD enclosures so there won't even be anything useful in the fire sale :(
I also think about all those cafés with free WiFi, where you buy a single coffee and take up a table for six hours while you browse Facebook and have your screenplay open in the background. I benefit, but the café doesn’t.
Can you elaborate on this? They mount in a standard rack and it looks like they have a backplane connected to a commodity motherboard. I'd think if you wanted it to be a JBOD you could replace the motherboard with an external SAS connector. Based on your nick I'm sure you've thought this through so I'm curious.
The only thing I see is that it looks like you have to slide the whole unit out to swap a drive, but that seems like an inconvenience rather than a show stopper.
Are you talking about their B2 product or their backup SaaS? The former has "fee that scales with usage", and the latter probably has enough normal users (ie. not data hoarders backing up 50 TB on the $99/year plan that they're not losing money overall.
$99/year with no limits on data, as far as I can tell. The context here is someone talking about personal backups, not Backblaze’s broader offerings. We know the market rates for cloud storage, more or less, since they’ve converged somewhat, across the industry. This makes it easy to calculate what kind of usage patterns would lead someone to save money at Backblaze.
Cloud services often have a free tier or cheap tier to attract new customers. IMO, this is not it. This is a product. These people are not turning around and signing contracts with Backblaze at work. At least, not many.
So it could be that people are just using it below its limits, or it could be that it’s subsidized by other parts of the business. Overall P&L is irrelevant—you want to be able to explain this product as a profitable product, as some viable sales channel, or as marketing.
My recommendation is to buy storage products where the company is making money off you, or nearly.
So do the calculations? As I mentioned in another comment, their unit economics are sound, with them making more than 50% gross margin.
>So it could be that people are just using it below its limits
That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
Is this some kind of personal challenge? It’s arithmetic for chrissakes, not calculus. S3 IA $0.0125/Gb-mo, ingress basically free, $99/yr÷(12mo/yr×$0.0125/Gb-mo)=660 Gb. Reasonable to ignore egress for backups—you may want to account for it, but it’s reasonable to ignore it. Do your own calculations if you have different assumptions.
That’s not hard. The numbers don’t have to be exact.
Maybe I just think it’s easy because I’ve run numbers like these for a living. Given X different storage configurations, under which conditions is configuration 1 cheaper than configurations 2 and 3?
> That's how many businesses work. Many low end gym chains operate on the assumption that most users won't use their services. There's no way that planet fitness can equip and operate a gym for $10/month per member, if everyone one of them went regularly. That doesn't mean it's a bad idea to get a planet fitness membership (assuming you'd actually go), or that they're at risk of going under.
The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
The economics of gyms are different because the cost is not only the membership, but also the time it takes you to get to the gym. Cloud services are a much more efficient market, which means the margins are much lower.
This includes AWS's margin, which could be quite handsome. Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Even taking the 660GB figure at face value, that's plausibly within the range of what I'd expect the average person to have backed up, especially when you consider that the pricing is per machine, and the standard desktop/laptop comes with around 500 GB of storage.
You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
>The gym closes I can buy a different gym membership. My cloud storage service closes, it may be difficult to move my data to a different provider, depending on how much data and what kind of rate limits. We’ve seen this before. People with data stuck in an “unlimited” storage service that’s shutting down, trying to transfer it but getting their egress throttled.
It's a backup service. It's not even trying to compare itself with the likes of dropbox or onedrive. You should have at the very least a second copy locally.
It could not be.
Storage costs are more or less race to the bottom. Storage business model is that it’s cheap and commodified, but sticky. Your data is stuck in S3 because it’s a pain in the ass to migrate. The margins come from non-commodified parts of the cloud (higher-level SaaS products) and I suspect egress costs (I don’t have the details).
There’s three parts to my understanding here. One part is my understanding of the business model, which explains why storage is a race to the bottom, why it’s not subsidized (or not subsidized much), why from a strategic perspective, the margins are getting eliminated. The second part to my understanding is the technical aspect—the calculations for how much it costs to stick hard drives in a data center and run storage systems on top with the desired accessibility and reliability, and I have some fairly deep expertise in parts of this. The last part is my personal experience working for cloud providers.
I can elaborate on the first two parts but for obvious reasons some of my knowledge is proprietary and I can’t share it.
> Bandwidth costs for the top 3 hyperscalers have converged to around 9 cents per GB, but nobody seriously thinks that's anywhere close to to the actual cost of bandwidth, or that hetzner/ovh is going to go under because they're offering 1TB ($90) of bandwdith for their $5/month VPS.
Bandwidth is not commodified like storage. I know that storage cost is close to operational cost because I’ve done extensive calculations and comparisons—and some projects are cheap enough that you have to revisit your theories about how they work on a technical level.
I don’t know why bandwidth is expensive. I’m not an expert on bandwidth or CPU or 90% of the other stuff. I only happen to have expertise on storage.
There are services you can buy where egress is the major cost and egress is significantly cheaper than egress from AWS, GCP, or Azure. Either egress is a complicated product or the margins are high. Probably some combination. Storage is less complicated and the margins are lower.
> You've also failed to address my point about their gross margins. The reliability of their books have been questioned, but even the short report didn't accuse them of cooking that metric.
I guess I don’t understand what point you’re making, or how that point about gross margins makes sense. It doesn’t make sense to me so I’m not sure what I’m supposed to respond to.
There's no way that planet fitness can equip and operate a gym for $10/month per member, if one of them simultaneously used all machines at all locations.
When O365 launched, they were using spinning disk for exchange. The issue was that they stranded capacity because of the IOPS needs of exchange. So “free”, (low iops) SharePoint and OneDrive for business data utilized that “free” capacity.
No - no exceptions.
We must, as sophisticated 21st century actors, insist that providers (especially providers of critical services) have financial interests that align with our own interests.
I don't care how big the parent is or how much money they have to burn - if it makes financial sense to keep you from storing your backups you need to go elsewhere.
Non-nerds are going to screw up backups without good UI. If you want to pay by the drink, like I said, use the hyperscalers.
If it was aligned with economies of scale, customers would get more storage for the same price every year.
We don't hear about cloud prices coming down that often.
How do you test your backups? You do test your backups, right? Your other comment mentions using dropbox, but that's hardly a real backup solution, and you could easily run into a situation where you need to retrieve files beyond the 30 day window.
And it’s such a good thing to remind people of. You don’t manage a backup service. Nobody cares about backups.
People care about restores.
I would personally be fine running tests on data that is recently backed up and only testing the data in glacier once or twice. Think about why you test backups in the first place—the main errors you’re trying to catch are problems like misconfiguration, backups not getting scheduled or not running, or not having access to your encryption keys. You can put your most recent backup in “infrequent access” and let older objects age out to glacier with lifecycle rules.
Glacier used to have really expensive retrieval costs. That’s now called “glacier deep archive” and as far as I know, major use cases are things like corporate recordkeeping / compliance (e.g. Sarbanes-Oxley). The costumers for deep archive should be sophisticated customers.
For most regular people, I suggest Google Drive or OneDrive because you get the value add of the their ecosystem. With Google, Photos are good. With Microsoft, the Office+OneDrive subscription is a great value.
Bandwidth costs can seem like a lot more, but when you purchase a 10 gig fibre, you have unlimited data, up until the full speed of the 10 gig fibre, 24/7. That number in TB can be calculated.
Clouds massively mark up services. Ones that underprice can have the opposite problem.
I have heard good things about the Backblaze service itself, and appreciate the hard drive reviews they put out.
On one side, the cloud is someone else's computer they will always charge more an more for it because customers have not learned the economics of cloud hosting and how profitable it is on the other end.
On the other side, a NAS is your storage computer, in a simplified home appliance form. Connect it to back up to any cloud you wish.
The company is being plundered and run for the benefit of the exec team printing shares for themselves. Nobody should be buying shares expecting the price to rise.
It’s bits in a barn, and they’re not making rent. That’s worried me enough to re-evaluate my relationship with them, especially in light of this research.
Financial statements indicate they selling services at 50% margin, but all the extra sales/admin/R&D add up to a loss
They could cut all of that I guess, it's not like object storage need a lot of innovation and they've already stopped making their own storage servers and
Gosh, if only there were a cloud storage provider that gave you an empty UNIX filesystem you could do anything you wanted with ... maybe something built totally on open standards that could handle hardlinks and symlinks and such ?
If only ...
It’s not the same service as backblaze’s client but it does everything I need, with dedupe
https://hindenburgresearch.com/gratitude/
Funny, I wrote this without seeing what the full URL was (I’m on my phone) so the use of the word “gracious” preceded me knowing the path was /gratitude
honestly I participated in the IPO then sold when it was up a bit - I had faith in their fundamentals as a business but they did not seem mature as a public company.
Doesn’t that seem like insider trading? I guess he can claim he didn’t know so and so was going to quit.
That’s part of why they usually deputize a brokerage to periodically sell a fraction of their holdings at intervals.
It’s been a long while since I paid any attention to this but I do believe I saw at least a couple of instances of an insider triggering weakness in their stock just by increasing the rate of sale of their holdings.
And Warren Buffet would use many frontmen to slowly buy shares without triggering a buying frenzy. You can’t just buy or sell 3% of a company without causing drama.
But… are there securities laws in this era where policy decisions are timed to pump/dump the lowest?
If you're a less technical CTO, Wasabi seems to do a lot more traditional marketing on things like sports games: https://ibb.co/21qNkM9b (NBA, NHL, Premier League soccer). I'm not saying that makes them better, but a lot of people with purchasing power are going to be seeing Wasabi around.
Wasabi has also been pushing value-add services on top of their storage with Wasabi AiR (https://wasabi.com/cloud-object-storage/wasabi-air). It's basically an AI-powered database of your content. If you're running a place generating lots of content, having a system automatically tag and make searchable all those images and videos is a really nice value-add. Being able to search for a video clip containing a specific person in your library junk pile of video can be nicely useful for a lot of organizations.
It could just be that different people get their brand awareness from different places.
Small data, or low risk tolerance: Tarsnap
We replaced with Borg.
- Wasabi
- idrive
Who else?
Cloudflare, scaleway, hetzner, OVH, Oracle, minio and dozens of others all have s3 compatible object stores
Most of them expect AWS have at least inter cloud transfer free (or discount Azure and GCP) being part of bandwidth alliance(BWA).
AWS S3 pricing is designed to keep you in the ecosystem , not be the cheapest, so tend to be costliest not cheapest unless your workload is archival.
For almost all non archival workloads egress costs will be on par or much higher than storage costs and AWS[1] charges a lot for any bandwidth.
AWS also makes migration really expensive so if you started with them, you are likely stuck as you would have to pay up to 10-12 months of cost to migrate out if you are on anything but the standard tier (cheaper the tiers higher the retrieval costs)
[1] All tier one providers have 10-20x b/w costs compared to tier two , however Azure and GCP discount for inter cloud transfers AWS does not
https://www.cloudflare.com/bandwidth-alliance/
It’s only Azure that isn’t on the S3 API train.
I vaguely remembered they announced it, but didn't find it on a quick search, thought maybe I was wrong.
> Initial Disclosure: After extensive research, we believe the evidence justifies a short position in shares of Backblaze (NASDAQ: BLZE). Morpheus Research holds short positions in BLZE, and Morpheus Research may profit from short positions held by others. This report represents our opinion, and we encourage all readers to do their own due diligence. Please see our full disclaimer at the bottom of the report.
Oh, and the finance employees refused to sign off on the books and a few are suing the company.
It sucks, but Backblaze is cooked.
But this is a hit piece. For any hit piece, I think it’s important for readers to know the motivations of the author, regardless of the validity of the claims.
(At least they disclose it at the bottom, but it should be in the title.)
this is activist short seller report/investigation, some of the points made:
- never been profitable
- execs selling aggressively
- loosing customers to Wasabi
- accused of cooking the books
- being sued by former execs for anti whistleblower/wrongful termination
- execs leaving
The very quick high level explanation is that in storage you talk about "stretch factor". For every byte of file, how many bytes do you have to store to get the desired durability. If your approach to durability is you make 3 copies, that's a 3x stretch factor. Assuming you're smart, you'll have these spread across different servers, or at least different hard disks, so you'd be able to tolerate the loss of 2 servers.
With erasure encoding you apply a mathematical transformation to the incoming object and shard it up. Out of those shards you need to retrieve a certain number to be able to reproduce the original object. The number of shards you produce and how many you need to recreate the original are configurable. Let's say it shards to 12, and you need 9 to recreate. The amount of storage that takes up is the ratio 9:12, so that's a 1.3x. For every byte that comes in, you need to store just 1.3x bytes.
As before you'd scatter them across 12 shards and only needing any 9 means you can tolerate losing 3 hard disks (servers?) and still be able to retrieve the original object. That's better durability despite taking up 2.7x less storage.
The drawback is that to retrieve the object, you have to fetch shards from 9 different locations and apply the transformation to recreate the original object, which adds a small bit of latency, but it's largely negligible these days. The cost of extra servers for your retrieval layer is significantly less than a storage server, and you wouldn't need anywhere near the same number as you'd otherwise need for storage.
The underlying file system doesn't really have any appreciable impact under those circumstances. I'd argue ZFS is probably even worse, because you're spending more resources on overhead. You want something as fast and lightweight as possible. Your fixity checks will catch any degradation in shards, and recreating shards in the case of failure is pretty cheap.
Very interesting. Could you name a few, am curious. I would be happy if erasure codes are actually being used commercially.
What I find interesting is the interaction of compression and durability -- if you lose a few compressed bytes to reconstruction error, you lose a little more than a few. Seems right up rate-distortion alley.
The economies of scale mean that you really have to have something like erasure encoding in place to operate at large scale. The biggest single cost for cloud providers is the per-rack operational costs, so keeping the number of racks down is critical.
[1]https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...
[2]https://www.usenix.org/system/files/conference/atc12/atc12-f...
[3]https://cloud.google.com/storage/docs/availability-durabilit...
[4]https://www.backblaze.com/blog/reed-solomon/
[5]https://www.usenix.org/conference/hotstorage13/workshop-prog...
[6]https://research.facebook.com/publications/a-hitchhikers-gui... they even do some interesting things with erasure encoding and HDFS
[7] https://blogs.oracle.com/cloud-infrastructure/post/first-pri...
While it's not "commercial", but CERN uses it and many other institutions.
https://indico.cern.ch/event/941278/contributions/4104604/at... --- 50PB
...
ah, okay, finally an AWS S3 presentation that mentions EC :)
https://d1.awsstatic.com/events/Summits/reinvent2022/STG203_...
XFS or ext2 (or 3 or 4 wo journal), without LVM or mdraid.
There's no point to adding RAID at the hardware or OS level for object storage boxes when redundancy exists at the application level. A drive with too many errors will be marked "dead" and just spun down and ignored.
Metadata servers OTOH tend to be engineered to be much more reliable beasts.
Can't seems to find anything specific about Wasabi uses ZFS on Google. And Wasabi doesn't change you on egress. So I guess they are similar in terms of pricing.
Although B2 seems to be way more popular on HN. I rarely see Wasabi here.
"The developers at Klara Inc. have been outstanding in helping Wasabi resolve ZFS issues and improve performance. Their understanding of ZFS internals is unmatched anywhere in the industry" - Jeff Flowers, CTO, Wasabi Technologies
https://klarasystems.com/
You could also search the OpenZFS repository for commits with the word Wasabi in them.
Wasabi only allows as much egress as the amount of data you're storing and I don't think you can even pay for more: https://wasabi.com/pricing/faq#free-egress-policy.
There's a number of CDNS that participate in zero egress.
https://job-boards.greenhouse.io/wasabi/jobs/4615087008
(Unless something’s changed since the last time I checked)
EDIT: OP is correct for CDN but if you use R2 even as a transparent copy from another S3 like provider it is allowed [1]
[1] https://blog.cloudflare.com/updated-tos/
If they're going to compress/decompress, ideally you'd want them to have that at the point of ingestion, then encrypted, and then store that on the target drive.
That way you can put very strong controls (audit, architecture, software etc) around your encryption and decryption, and be at reduced risk from someone getting access to storage servers.