And sometimes the factory building whatever has a giant machine that can easily do what would take you quite a long time to do by hand.
It's been a valuable lesson for myself to question just how much of my internalized "obvious" knowledge about technical matters is apparent to clients and partners. Never assume.
If the low bidder gave that bid because they didn’t notice, then also lulz - tends to weed out the problem children. Really painful for the customer and said contractor though.
We had a pond put in a number of years back. The guys that did the work laid out what they would be building, showed us everything, and we agreed on it. While digging out the earth for the pond, they ran into a boulder not too far underground. They spend about half a day fighting with the boulder, seeing if they could break it up, etc; whatever they could do to get it out of the way.
When we went out to chat with them and see how the work was going, they told us about the rock and how it's being difficult. We told them we'd be fine with them changing the shape of the pond to just work around the boulder. They were happy about this because it made their lives much easier.
The cost of the creation of the pond was set before they started work on it. They gained nothing by trying really hard to create what they said they would; other than the fact that they did a good job and had happy clients. But they spend the time trying to deal with the problem without any real complaints (the talk about it being difficult was a casual chat, not them coming to us to complain).
You saved them a lot of hassle by being flexible, but were under no obligation to be.
Smart contractors build in the odds of this happening in their contracts. Dumb ones expect everything to go right all the time, and then try to run away or cheat when it doesn’t. Or lose their shirts.
This is technically true, but misleading. They have to follow the contract, but the contract likely includes exceptions for unforseen issues. And while you can technically sue a contractor for breach of contract the only reason to do so in this situation would be spite. The rational choice if a contractor cheats you on a small project is to be sad and move on. (Google around and you'll find lots of people who learned that the hard way).
That decision would have had to be before they signed the contract.
No one knew about the geology.
At the point they did, it was too late.
It was great they didn’t try to avoid their responsibility, many would have!
"An amateur will build what you want and an expert will build what you need."
Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands.” ... “The fewer moving parts, the better.
There are plenty of contractors who know how to deliver an optimal solution and who make incredible money.
One way to view it is that they're each targeting (deliberately or unconsciously) a different market-fit, with respect to cheap/good/fast.
For some jobs you might want the advice of the first "compromise this and it'll be cheaper and faster" group.
When working on one particular unit, I found that they would order a bunch of plastic panels, cut to size, and some with precut holes. And others with secondary drawings, instructing us how to drill out a bunch of holes ourselves. Sometimes not even all the holes, just a few, in addition to the precut.
When I asked why we didn't just ask the supplier to cut out ALL the holes, I basically got a shrug.
So I merged the secondary drawings and sent them to the supplier, and the assembly team never wasted time on that nonsense again.
Then I looked around and found a company that would provide me the correct length of wire cut to size with the right terminals crimped in place all for less than I could buy just the bulk wire in the first place with no setup costs. Minimum order, IIRC was 100 pieces, which is peanuts.
Blew my mind!
At least I only struggled for a month or two before realizing that there had to be an easier way :-)
For a while, I thought that maybe an exception would be having a technical writer on staff but after reading this post I'm significantly disheartened on that front too. I'd be interested to hear if anyone on hacker news has experienced good internal documentation and even more interested if any of you folks have experienced anything truly _great_.
For my contribution, I've found that documentation fails for a couple of reasons. The first is the burden of correctness. The people who most would like documentation are also the people who most need the documentation and they usually are also the people most likely to be reluctant to contribute to the documentation because they don't feel they can accurately represent the information. Imagine someone ramping into a feature and spending a few days reverse engineering how it works, collecting info, etc. Sometimes they'll put it up somewhere but a lot of the time contributing partial information feels 'wrong'.
And the second bit I find to be a big reason why documentation efforts fail is just the sheer friction of putting it into the documentation store to begin with. In confluence, for example, if you have a bit of information it can be tough to work out how to categorize it, where in the hierarchy it should go, etc, etc. Or if it's a GitHub wiki you want to put it somewhere that it is discoverable but also be careful that it's 'correct' because you don't want to break backlinks if it gets recategorized.
I've mostly given up on it at this point. Instead, I take detailed personal notes and make them publicly available. It doesn't have to be correct because being advertised as personal notes means that it's my opinion on the truth rather than objective fact. It isn't far away from my codebase, I can just tap a short keybinding in my editor to type my notes or search them and I can link directly from the notes to lines of code in files to jump back and forth. The particular system I use means that if I write a short snippet of code to solve a one off issue (like calling a path helper to derive a URL that I can't find in the interface) I can even drop it in a code block and execute it right from my note-taking tool. It isn't ideal for sure but I've gotten way farther in having a shareable knowledge base this way than I have in literal years of trying to get a shared, useful documentation store spun up.
It's just not possible to do this. I think it would be better to talk about "effective documentation".
My opinion on the matter:
Three levels of documentation
1. high level documentation that describes the problems and the goals from the business perspective
2. mid-level documentation that describes the architecture that gets us there
3. low-level documentation, aka code.
high level gives you direction (where), mid-level gives you context (what), low-level gives you implementation (how)
You need the what to understand the how, and you need the where to understand the what.
The other piece is an acceptance that you can't document away the need for tacit knowledge.
To draw an analogy.
No amount of documentation is ever going to allow a kid to hop on a bike and ride it perfectly the first time. That is not what it means to teach a child to ride a bike.
Instead the goal is to have enough documentation to minimize the amount of time it takes that child to gain the tacit knowledge necessary to ride a bike acceptably.
Once you accept the above and lower your bar, "effective documentation" becomes much more achievable.
In my team, I have started spelling out more concretely what I expect from the documentation at various levels. Keeping with the three levels specified by the GP, we always document:
1 - business rationale: what are we building, and who are we building it for (enumerate the stakeholders and their processes)
2 - component diagram: a graphical sketch of what different building blocks are involved (datastores, api endpoints, webservers, etc), and showing the direct connections between them
2 - data flow diagram(s): showing entry, exit and storage points for each class of data moving through the system
3 - starter documentation: what does a developer need to know/do before working on this project? This includes tooling, pointers to outside documentation, and a reading guide pointing to the rest of the project's documentation.
The rest of the low-level documentation is the responsibility of the team itself.
By insisting on documenting the second layer mostly graphically, there is less potential for overlap between the different layers. Of course, no documentation system is perfect, but I find the above has suited us quite well for what we do (we write business integration software, no huge client applications or software suites. YMMV, obviously).
How do you keep the layers consistent? How do you keep track what update to some layer is not covered in some other layer yet? Is that visible right in your layers? or some outside backlog?
There are no airtight rules other than being mindful of your own processes. Our workflow generally looks like this:
Changes to level 1 documentation are done by the product or project manager in preparation for the work, so it's pretty easy to make that part of the documentation lead, rather than lag, the implementation. It's stored in the project wiki, rather than with the code. But that wiki is still git-based, which means we have an easily auditable changelog to check when a particular piece of information was last updated.
Layer 2 changes are usually done by (senior members of) the development team, but it's done in the week before the implementation starts. Since we use those diagrams to communicate with the rest of the development team about the changes required, the documentation updates happen rather organically.
I find that layer 3 documentation updates tend to be the hardest to capture in a process, because the code itself is much more in flux than the higher layers. What helps us here is that our developers know that the codebase they're working might not be worked on again until months in the future, so it's in their own interest too to keep the documentation correct. That said, we do have an explicit check for documentation updates in our project delivery checklist.
The most challenging situation is when we discover during development that an architecture redesign is required, but most of the time an architecture adjustment also means we need to re-evaluate the workload and possibly the delivery date, which basically means our process resets and we re-involve the product or project manager. Funnily, this then usually results in more documentation rather than less...
We sometimes also have a 2-person development team for small changes (never 1 person though), in which case all three layers are handled by the same people. But because they're updated in different stages of the project, and because they're already used to the same workflow in larger teams, the small team size doesn't affect the documentation quality as much as I initially feared.
A tool that tells you who commits most frequently to a service or file is better than an outdated yaml file with an owner. A firewall that won't let your service make outgoing requests unless you list its dependencies is better than a manual list of dependencies. An API doc compiled from the type definition of your API or from a schema which requests are validated against is better than a manual API definition.
Sometimes there were people who didn't document coz they kind of liked being the font of all knowledge.
Confluence also sucks - even harder than jira, if that's possible.
I'll settle for examples.
Documentation is a multi-headed hydra where you have different levels of technical competency and foreknowledge and familiarity with the programming environment patterns in use the organizations, particular technical stack organization, the people inside of it, what the product does, etc etc etc
In particular, every GitHub wiki belonging to an organization or a public project itself is a git repo of markdown files that you can clone and commit to. I probably would use that for per project documentation w/ a separate folder for managing broader notes. I think it's important to separate the markdown from a project's code because otherwise you can have important markdown updates trapped in a branch that hasn't been merged yet and because git works by line rather than by word you can get into complicated merge conflicts by features that touch similar business concepts.
Org mode happens to add some niceties on top in that you can have an example code block (roughly comparable to a triple back-tick code block in markdown) that also can be executed. The example code block can take some arguments like what interpreter to use and will paste the output into another code block underneath the source code block.
Because org mode is also used for organizing tasks it is easy to use it as a scratch pad as you grind through tasks and if something turns out to be useful you can promote it to a top level note (or org-roam node). And even if it's not obvious that it's a note candidate, just thinking through problems 'out loud' in org means you can search and find it later when the same or an adjacent problem comes up.
Another favorite feature I like in org mode is that in any code at any time I can tap out 'SPC n l' and it will capture a reference to the file and the line in the file and allow me to link to it in my notes. It doesn't capture by line number but by copying the literal line, which means that as feature updates push the line number up and down, it still will be able to find exactly the right file in exactly the right line as long as the line's content hasn't changed. It also serves as a rather nice canary because if the docs link to a line that no longer exists, then the documentation has probably gone stale.
The last major win with org mode vs markdown is with org-roam, which borrows the ideas from the standalone program Roam Research (which also is very similar to notion). Every 'node' you make initially is a file and you can fill it in as a wiki type structure. Everything is flat on the filesystem and has a unique id prepended to it so you can't overwrite it with another node of the same name. There's also a UUID associated with each node so if you move from one place to another, none of the links need to be updated. Contrast that with a regular wiki or markdown where when you move a file to a new folder you either have to leave a redirect behind or go through and update all of the back links.
You can also start a node as a subheading and later on promote it to a full file with all the backlinks still working correctly.
I started using org mode years ago with spacemacs, got frustrated with spacemacs, and gave up until earlier this year when I found doom emacs and gave it another spin. I wish I had a better recommendation for an alternative but after looking for 4-5 years for something as good as org mode, I could never find anything. Notion seems like a popular alternative but there's also a lot of shiny bells and whistles in it that can cause 'productivity procrastination' where you spend time configuring and turning knobs instead of actually using the system. Ironically, the second best tool I found was plain pen and paper using a minimal bullet journal technique and taking special care to do indexing. Of course, you can't share plain pen and paper in that way, but the name of the game for personal use is low friction and ready access. The better a system scores on those two in term of note-taking the more useful it will be.
Get in touch if you ever want to discuss Backstage. Email in bio.
SSO is supported on all plans by default.
For deals bigger than 150 devs, we have found customers prefer to talk to sales.
The plugin system isn't bad - but its a false narrative to believe you plunk this down and get some sort of valuable developer portal at an enterprise level (small biz sure - fortune 500...yeah no).
Spotify does open-source a number of their plugins, but after having looked extensively at which ones they make available and which ones we would have to write (and deal with their plugin system for) we opted out. The cost/benefit ratio is definitely there for a technical writer, but IMO its not there for serious development. (Pls - Anyone who disagrees with that - begin by telling me why you don't use Joomla as a base for all your custom applications )
Team integration at a company level is another challenge with Backstage. GitHub (the basis and sort of harvested DB for BS) is largely disconnected from company structures and AD and all the things for most companies. Yes GitHub enterprise is a thing... no in general its still not connected to company structures. There are teams in Backstage, but they are generally managed in isolation from the rest of the company which presents its own challenges and integration problems with other systems. Don't even get me started on the fact that asking everyone to put their service defintions in specific files in specific places in GitHub is no better/worse than making them enter it into a ui or spreadsheet or confluence article or anything except now its scattered in 10,000 repos - good luck getting everyone to change that.
Ultimately this isn't software that you simply install and use. Its a base that will likely require extension, and doing so means going down a one way path of adoption. Nothing it does is compatible with anything else. Is that good for Spotify - sure. Lots of folks helping build software they want to use is great. Is it good for the community? I'm not so sure. This article felt like a paid add (not suggesting it was - just saying it read like one) more than an honest plus/minus of the software and is anyone really surprised a technical writer couldn't build a proper enterprise developer portal by themselves? In other news - water is wet, the sky is blue and my kid wants to watch youtube.
I used to work on a consultancy (https://frontside.com) who's specializing in adopting Backstage at Fortune 500s like HP. When you adopt Backstage at scale, you rely on processors that scan your LDAP, GitHub, etc to discover, stitch, and feed cohesive data to the software catalog. HP is giving a talk next month on how they manage 100,000+ entities in Backstage.
Backstage does a lot of heavy lifting that I'd rather use than have to build from scratch.
I'm being a little silly there but my issue isn't that this isn't valuable software in the right situation. My issues is that its presented as some sort of magic bullet for a specific documentation issue when the problems described are generally not software related or software solved.
Most of what is presented in the article isn't about BS doing what this person attempted to do for 2 years. Its about switching paradigms and letting folks document in their repos instead of in a centralized system. One could just as easily look over the repos in a github org and go to those files in a repo. Both solutions require standardizing where things are documented..... so does a centralized system.
The big callout should be that when trying to centralize docs that align to apps they become disconnected from the apps themselves and thus its a pita to get everyone to do everything in 2 places. I'd be very curious if they completely turned off BS if there would be any loss... if not why bother with the overhead. Templating and other things are well handled by MS Clis, cookie-cutter or other more portable concepts and who needs yet another team and resource mgmt system. Splunk and other off the shelf systems can easily provide more comprehensive and robust stats and generally speaking there is a more enterprise level solution for just about everything it does.
Yes BS is a swiss army knife but when was the last time you saw a chef use one of those in the kitchen?
With that said, what's the low hanging fruit to entice adoption of backstage into an organization that already has a "process" like mentioned in the article of whiteboard meetings with stakeholders. Obviously that's not a great use of time, but it's also no extra learning or work, that SME can talk about their function without any preparation and can interactively answer any questions the learner asks at the time.
That doesn't scale in practice, but it also doesn't require any more whole team buy in
If there is a lot of legacy in your org, or if there is not enough buy-in then it will definitely be "yet another" system. There's a lot of functionality to unlock once you have a bunch of data in Backstage, though, so that could help tip the scale. The cloud costs dashboard that helps to show how much every service costs is a good demo :)
One benefit to Backstage's approach is that you don't necessarily need to change your processes and tooling to work with Backstage. It can help, but it's not a strict requirement. Other tooling in the space requires you to buy in to their specific workflows and so on whereas Backstage is a bit more agnostic.
Solving this problem is why I started OpsLevel  a few years back. Everyone always starts with a spreadsheet because they're flexible.
But spreadsheets quickly fall apart because they're not complete / up-to-date, automated, or really scalable.
We figured we could automate a whole lot of collecting service info from different places (Git, k8s, other deployment streams) and keeping it up-to-date.
 - https://www.opslevel.com
It's always a tricky balance, but I definitely sympathize with the feeling of just getting started without having to talk to a salesperson.
“Build an ecosystem, not a wilderness"
which is ironic because a wilderness is arguably the most complete and perfectly functioning ecosystem you're gonna get.
Build a wilderness, not a parking lot
Build an ecosystem, not a science project.
Build an ecosystem, not a microbiome.
To be clear, Backstage is not an alternative to Gitea, Github or anything like that. It's not a source code repository hosting tool.
It is not a turnkey solution per se and what they call plugins aren't plugins in the traditionnal sense where you would just download a package and put it in a directory. It requires editing the actual react code to set most of them. So maintaining backstage involve maintaining a set of patches and a pipeline to build a docker image.
One of our team members who previously ran a Backstage instance at $OldJob wrote about that pain:
Will checkout OpenContext, thanks for the link
I can't seem to find a data catalog that lets my tie information across databases + APIs + Kafka topics. Each catalog of schemas is separate with no way to link them really.
Edit: This is a serious question. What is that value prop of backstage over confluence?
Unfortunately, given their affinity for testing things, the How To docs starting taking on that QA perspective and everything ended up in a QA "space". Sure, it's easy to move them around, but there wasn't a clear cut way on how to separate the "How To" from the "QA" - and then they left.
Since then, people add things when managers say "make sure that's in Confluence" and it ends up getting placed in the areas they're most familiar with, not necessarily where it should be. Add in the aforementioned lack of standards and no two documents look the same. Then we started using Confluence for devs to write Test Plans and QA to execute them - and now it just makes my brain hurt whenever I have to go in there.
Some of it is a cultural problem: we're email and Teams heavy and so much of our "documentation" is buried in those tools. So then we adopt something like Confluence to move our knowledge base to that, but finding stuff in Teams is a nightmare (posted in another thread on that). So now rather than having one tool to rule them all, we just have another half-baked attempt.
We're monolithic for the most part, so I'm not sure Backstage would help us, but the article mentioned Stack Overflow for Teams, so maybe that would? (insert relevant XKCD here on competing standards)
The stuff that's not autodiscovered can live right next to your code, in any repository, so devs can just update some YAML keys when they're making other changes.
You can lint the backstage changes as part of PRs
Also I've never ever seen developers consistently keep stuff up to date in Confluence.
There's just so much friction in that interface.
There's a wide world out there, which some HN posters "forget."
Edit: Thanks everyone!
It has more features including onboarding, access control, etc
(Disclosure — I’m an investor.)
Backstage is super configurable, but requires that you build your own developer portal. That generally requires a dedicated team of people at your company to think about dev workflows to support, building plugins, plus hosting/maintenance.
At OpsLevel, we're focused on a much more turnkey solution. We want to get you up and running in under an hour, not weeks / months.