Maintenance programming for pay can be rewarding iffffffffffffffff management has a deep understanding of and appreciation for the task. There can be a lot of joy and satisfaction in adding tests, fixing edge cases, fulfilling user needs, "quality of life" improvements, optimizing. Honing.
My experience is that this is almost never the case.
Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.
Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.
But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.
obviouslynotme 303 days ago [-]
It's worse that that. Most companies give career incentives to new projects. Google is probably the most well known offender, but they aren't the only one.
Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.
chii 303 days ago [-]
> the most valuable asset of a programmer is his mental model of your system.
Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.
Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!
Cthulhu_ 303 days ago [-]
The problems with documentation is one, organizing and keeping them up to date is another, and... actually reading it is the other thing. I'm guilty of that myself, because I already assume the documentation will be out of date. But so does everyone else, thinking it's pointless to write something down because it's easier to just ask someone else.
obviouslynotme 302 days ago [-]
If management cared, not updating documentation with every merge would be a strike, with multiple offenses leading to firing. They don't care. All large development is just slapped together and shipped. By the time the reaper of long term problems comes, the problems are too big to fix.
atoav 303 days ago [-]
We can also break it down even more, managers are only allowed to pick one of the mutually exclusive options:
- Well documented, well maintained code that everybody could work on
- Quickly finished projects that nobody ever touches again, but only the programmer(s) have a mental model of it
The first option is more expensive at first, but way less expensive if you happen to have to rewrite it from scratch at some point (and if you are not maintaining and document it, that moment will come and it might not come at a point in time that fits the schedule).
The second option is less expensive first, but it will bite you in the arse if the programmer(s) are gone or forgot how it is done.
Everywhere else managers try to minimize risks, here they often just don't aeem to get the trade-offs involved.
chii 303 days ago [-]
The missing bit you forgot is that even if the first option was chosen, it very quickly devolves into the second option, whether the managers like it or not.
hinkley 303 days ago [-]
Google has turned it into a sport though. That deserves special mention.
civilitty 303 days ago [-]
They’ve even got their own mascot. It’s a gravestone.
hinkley 303 days ago [-]
Are you being serious right now? Pics. I need pics.
I've got mixed feelings about this. Google's frequent "killings" have caused me to never trust them or want to buy in to their products.
But at the same time I think that it is good to "honor" "dead" products. Those products encompassed millions of hours of blood and sweat and toil and so on. I don't know the best way to memorialize that but sanitizing them from history entirely ain't it.
hinkley 303 days ago [-]
A lot of us got there a few years ago. So the perverse incentive is that many of the people who would shun them have already, so why not keep misbehaving?
ern 303 days ago [-]
I was almost bounced by HR during a screening interview a couple of years ago when they found out I had been doing maintenance work for the previous six months. I had to explain that it was a tiny part of my overall work experience. The hiring hypocrisy is strong.
Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.
JohnBooty 303 days ago [-]
That's absolutely mental. I know maintenance work is uncool and kind of a career... well, not a stain, but it's not glamorous.
May I ask what sort of company this was? Some "unicorn" startup? Stable oldcorp?
jckahn 304 days ago [-]
Agreed. Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money? It seems much more sensible to leave the working code alone and focus on new revenue-generating opportunities.
fsckboy 303 days ago [-]
> focus on new revenue-generating opportunities
in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.
thfuran 303 days ago [-]
I think that’s overstating things a bit. Even in mature industries, new technology can change the field. But it certainly isn’t all about a hyperfixation on growth.
bbbobbb 304 days ago [-]
That's not what the post is about. They describe the process as mostly fixing issues with existing software, so the motivation is that something is broken or not working as required anymore.
Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.
JohnBooty 303 days ago [-]
[Refactoring] is not what the post is about.
Not directly, but kinda, at least in my experience.
Clients and management generally understand that old code needs to be maintained. They don't love that reality, but that's not generally a tough sell unless they're completely clueless. And they're usually not completely clueless.
The tension in my experience comes when an engineer wants to perform something other than the fastest, sloppiest, filthiest duct-tape maintenance imaginable.
Management accepts the need for fixes, but often does not have the technical/domain knowledge to understand the short- and long-term merits of Proposed Fix A versus Proposed Fix B. And naturally, incentives heavily push management toward the quick fixes by default. Time is money, after all. And managers are expected to meet quarterly and annual OKRs. We don't have a good way to quantify the long-term value of a "proper" fix versus a "duct tape" fix. And so on.
So for me maintenance work is difficult to separate from notions of longer-term software maintenance such as refactors, dependency upgrades, etc.
robocat 303 days ago [-]
> Refactoring
I personally don't believe refactoring should ever be part of "maintenance".
Maintenance needs to avoid YAGNI and avoid the engineers natural desire for over-engineering and unnecessary perfect polishing and abstruse abstraction.
Refactoring is something that should be driven by new features (or worst case towards rewrite), and few have the skill to do it well.
I'm not arguing the sibling strawman that maintenance be "the fastest, sloppiest, filthiest duct-tape maintenance imaginable."
Excelling at maintenance is highly skillful work and often poorly rewarded.
JohnBooty 303 days ago [-]
Well, I certainly agree that engineers tend towards YAGNI and this must be avoided.
Refactoring is something that should be driven by
new features (or worst case towards rewrite),
This overlooks a whole other class of factors that can drive maintenance: changes in the environment.
Assuming we're talking about a program that does something useful and isn't just a screensaver, it's probably ingesting data from the outside world. That data can change in a variety of ways. API partners and users can introduce changes on their end and will eventually find new and annoying edge cases. Or maybe the volume of data increases by 100x and changes must be made on your end.
Or maybe your software sort of rots as the security vulnerabilities pile up. Eventually you've got to make major version or point upgrades to the framework and/or various dependencies.
We want to handle these demands judiciously and as a rule, yeah, refactoring isn't generally going to be the answer but sometimes it is.
dragonwriter 303 days ago [-]
> I personally don't believe refactoring should ever be part of "maintenance".
Of course it shouldn’t, because software maintenance does not exist. Programs don’t degrade with use. All “maintenance” on software is, like new development, the implementation of new context->behavior combinations, and calling it something different is a result of applying an inappropriate physical-product metaphor.
JohnBooty 303 days ago [-]
Of course it shouldn’t, because software maintenance
does not exist. Programs don’t degrade with use
This is some real pedantic gymnastics and you're not even correct.
The word "maintenance" doesn't imply some kind of physical product metaphor. Sorry. That's not what the word means.
SideburnsOfDoom 303 days ago [-]
> software maintenance does not exist
There are several activities in software that are similar to maintenance of physical products. e.g.
* that parsing library that you used 2 years ago and was in good working order then, is now outdated and has known vulnerabilities that are not present in the latest version, apply update.
* Those cloud-hosted databases that you use are an older version that the cloud vendor has deprecated and will some time withdraw support for, apply migration to a later one.
riku_iki 304 days ago [-]
> Imagine you’re a CEO: Why on earth would you want your expensive engineers reworking something that already makes money?
CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.
JohnBooty 303 days ago [-]
True, but app performance/stability metrics are (often extremely laggy) trailing indicators of code quality. So by the time the app starts falling over, things are extremely rotten, and it's not necessarily correlateable to any specific maintenance decision(s).
tangjurine 303 days ago [-]
By the time the app starts failing, usually the people who wrote it got burned out and left :/
esafak 304 days ago [-]
Which happens if the CEO is conscientious enough to insist on collecting and looking at those metrics.
pixelrevision 303 days ago [-]
Heh. Going to be a lot less maintenance work if they are.
whateveracct 303 days ago [-]
CEOs tend to be short-sighted when it comes to engineering management, that's true.
chii 303 days ago [-]
and yet the CEO would not find it ridiculous to spend money maintaining plant and equipment in the factory.
It's just a lack of understanding. They imagine they understand, because software doesn't get worn out, and doesnt degrade obviously!
303 days ago [-]
juliangamble 303 days ago [-]
Security and risk. You can run several different code/pom/package.json scanner tools that will articulate a risk that will send your IT security department into a panic attack, which you can sell as time you need to spend fixing things.
dagw 304 days ago [-]
It is not at all obvious that trying to develop new revenue-generating opportunities will actually bring in more revenue long term compared to optimising and improving existing revenue generating sources. And it's not like the old code stops working once you start reworking it. Give your engineers N months to rework the old code, and if it's not better you don't have to deploy it.
Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.
hinkley 303 days ago [-]
Why would you turn off manufacturing equipment to do routine maintenance? It’s making you money when it’s turned on, and not when it’s turned off, after all.
varjag 303 days ago [-]
Did ball bearings in application's garbage collector wear out?
As an embedded dev I have thousands of devices that were commissioned and are on 24/7 duty. The oldest still running fleets I believe date back to 2007. Some of them had firmware updates, but many of them (despite numerous bugfixes and improvements since) run on original software. Customers are reluctant to update if there's no specific need arises in their static operational environment.
Naturally in business logic world the requirements are never as static. But when they do change you can likely sell the management on it as necessary development.
hinkley 303 days ago [-]
The “seals” wear out all the damned time, which is basically the same thing.
throw239487 303 days ago [-]
Does anyone have any tips on this beyond the additional "sales" type stuff where you have to put in a ton of extra effort to measure and sell your work?
I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.
I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.
JohnBooty 303 days ago [-]
I think one of the keys is consensus-building. If one engineer tells management that it would be a great idea to do XYZ, management doesn't necessarily know whether to believe them.
But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.
hooverd 303 days ago [-]
Nobody cares about "eliminated a whole class of tickets" aside from the support staff, and really who cares about the support staff at most companies. What would the phone monkeys know about our software. /s
jebarker 304 days ago [-]
I'm currently on a sort of sabbatical from greenfield research projects. I'm doing maintenance programming on a large widely used open source project. It's been really refreshing. I didn't realize how much I was feeling the pressures of 15+ years of greenfield research. By contrast the maintenance programming is clearly defined, easier to put down at the end of the day and unambiguously useful. I'm also rapidly improving in my debugging, code understanding, profiling and optimizing skills which will make me a better programmer when I do return to writing new code.
aidos 304 days ago [-]
I’ve described this to other programmers before. Greenfield projects are fatiguing because you still have options—too many options. Every layer is still open for you to rethink and rework. Let the hand wringing commence!
As the months and years wear on though, things change. Each bug has an obvious solution, and you’ll take the path of least disruption. Implementing new features is a matter of figuring out which trade offs you’re most ok with. Maybe you need to refactor something first, but that’s an obvious path itself.
You’re painted in a corner. A beautiful corner where the only safe thing to do is to tread carefully. But you don’t mind, because you know where you are and what you need to do to get to where you want to go.
jebarker 304 days ago [-]
Nicely put. I'd also add that for me being a researcher became my identity and so my self-worth became tied to the perception and adoption of the research. With an established project (especially one I didn't write) there's already a stable acceptance of it's value and so I'm free to just maintain and improve it.
SKILNER 303 days ago [-]
I gave a presentation many years ago to a software process group in Silicon Valley entitled, "Unsolved Problems of Software Maintenance". The problems are all still there.
Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.
I heard this first from Joel Spolsky, "it's easier to write a program than read it."
The article uses AS400 applications as an example, but is general enough that it applies to applications in most languages.
This is a field begging for help from AI.
hinkley 303 days ago [-]
Someone needs to start doing comp-lit style studies of computer code instead of literature.
Most of the problems of API design could be remedied by people stepping through their own code and looking askance at the whole process.
x1f604 304 days ago [-]
Do people understand code by line-by-line reverse-engineering what the code is doing, or do they understand it by relating it to what they've written before?
If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
For system design too - if you've designed lots of systems yourself, then when you see a new system, you'll be able to relate it to the systems that you've designed before.
So maybe building greenfield projects also makes you better at maintaining existing projects?
It'd be great if someone could point me to some existing literature on this topic. I've looked around and can't find any.
tivert 304 days ago [-]
> So maybe building greenfield projects also makes you better at maintaining existing projects?
I think it does, because it builds a higher-level sense of how something "could" or "should" be and familiarity with thinking at the system level.
I've had a lot of problems with people who (seemingly) only have experience maintaining projects. They seem to have a tendency to focus narrowly on "fixing" a bug close to its source, and often lack understanding of a lot of fundamental technologies (because they only attend to things that have broken), and get stuck by a kind streetlight fallacy where they fix things in more familiar areas. The end result is ugly kludges and whack-a-mole bugs where a superficial "fix" in one area causes a new bug to pop up in another.
ye-olde-sysrq 303 days ago [-]
>> So maybe building greenfield projects also makes you better at maintaining existing projects?
Only to an extent, i think.
> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
I think "i've never bootstrapped a project before" is easier to cure than "I don't have a good sense of what's expensive vs what's cheap tech debt."
Some tech debt is basically 0% interest. Was this small hack bad? Yeah. Will it need to be fixed, eventually? Yes, for sure. But does it compound every day, or does it just sit there being a little yucky? Very easy to determine in hindsight. Very hard to predict as you write the hack. The end result being the simultaneity in my example. People will over-engineer things that won't end up being problems and under-engineer things that will turn out to require compounding hacks and it would've been cheaper to just get it right the first time.
65a 303 days ago [-]
This. I've worked in both greenfield and brownfield areas. The chief failuremode of greenfield is that it works, but it is unscalable and shoddily done. The chief failuremode of brownfield is some kind of feature cardinality explosion that becomes impossible to maintain after some critical juncture. Reading between the lines, they're sort of the same failure mode happening at different times, but both driven by results-driven-programming rather than architecture.
tivert 303 days ago [-]
>> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
> And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
The best thing is to build a greenfield project and then maintain it for several years, fix your mistakes, then do the same thing on a new project again (including the maintenance).
You really need the full spectrum of experience, and put yourself in the shoes of the future maintainer when you're building something new.
rvba 304 days ago [-]
There are lots of greenfield-only programmers who build something (by glueing stuff together), then they proceed to ship their barely working software to get that promotion / bonus and they run away to the next project
They never have to maintain anything, some poor soul has to fix the broken software.
They also dont even know what mistakes they made, since they arent there to solve it.
hinkley 303 days ago [-]
I think there’s something to be said for taking a job on a late stage project early in your career, participating in RCA analysis of all of the problems, and then picturing yourself in the room when those decisions are made and asking if you would have made the same decisions or better ones. Then work next on a greenfield project to test your theories out and adjust/grow.
I never felt like I had 1 year of experience 5 times because I moved between several kinds of projects at different lifecycle phases and with different constraints, and drew lots of parallels between them. At the end of five years I had project histories in my brain that spanned more than ten years. And got jobs that should have gone to someone with 8 years’ experience. I do not think this was a coincidence.
fatnoah 304 days ago [-]
I haven't had to do it a lot, but in previous cases where I came into a codebase blind and had to fix bugs or figure out what's going on, I've always started by mapping the flow first and then digging into the details of what's going on. When looking at the flow, I go breadth-first through everything and then dig in from there.
I maintain two major maps, which are what is the code logically trying to do, but also what exactly it's doing. (i.e. Sanitize inputs, check error conditions, do work, return result as a logical model and then "ensure foo is < 0 and > -10" etc. as the specific checks).
darkhorse222 304 days ago [-]
>If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
Let's hope what you write the first time is a good practice.
swader999 304 days ago [-]
The joy of Greenfield is short lived. At some point, it becomes too costly to fully refactor and you are stuck with what you chose and edging closer and closer to legacy dev.
Some projects though just keep on trucking along and doing the next part with the newest shiny thing. You end up with five ways to access data, three ways to send emails and fifteen different methods of report generation. Frankenstein architecture! This architecture also emerges when you have leaders that don't make decisions and peeps debating all day on slack and then doing their own thing anyway because they didn't really want to offend anyone and everyone's ideas are great.
gregjor 303 days ago [-]
Author of the article here. Thanks @pramodbiligiri for posting. Surprising and humbling to see this old article on the HN front page. I haven’t read it in years.
When I wrote that, in the wake of the 2008/9 meltdown, I intended to give programmers some career options to think about. More interesting and stable work out there than FAANGs and startups.
vsareto 303 days ago [-]
>Skilled programmers recognize what makes good code good, and what makes bad code bad
You're still somewhat biased, so you can get put on projects where the style is different from yours. If you happen to think that style is bad, it won't be fun to work on.
You shouldn't want to change the style. Chances are you'll do it incompletely (job change) or the time spent won't be worth it. If someone else comes along and doesn't like your style, we're just going to repeat this process.
There's been way too many different approaches that work for developers to be unified on what styles are best.
Rexxar 303 days ago [-]
Style is not what make code good or bad.
lodovic 303 days ago [-]
I'm still hoping someone will create the ultimate linter that can format source code in your desired style on checkout, and formats it back to the repository's style after a push
hiAndrewQuinn 304 days ago [-]
I love maintenance programming. I love legacy programming too. I have a story where the two worlds collide.
At my last job I had to work with a PowerShell codebase that `cloc`ed in at over 100,000 lines, and was absolutely ancient to boot, having been added to in bits and pieces by sysadmin after sysadmin over the course of 12 years total, in 3 different human variables-and-comments languages, all of whom were Windows power users with very clear ideas of what needed to be done, but also all of whom were very clearly learning to program for the first time, at least in the wonky PoSH paradigm. AKA: Classic big ball of mud situation. Every new customer had to have their details run through this codebase, however, and I saw the potential to eventually boil this down to a simple 6- or 7-parameter cmdlet (PoSH for "shell script"). I got my undersized chisel and toy hammer and got to work.
In some ways starting work on the proj felt more like beginning to dig into an ancient COBOL database in a System/360 mainframe than anything resembling the state of the art in 2023, or even 2011. There was a single VM that everything had to run on, and the script couldn't even be called remotely via SSH because the only way in was via a web browser based remoting session. The connection was spotty and would often drop halfway through coding in there in my dinky little Notepad++ session, requiring minutes of reauthentication just to get back to work. Unit testing (which PowerShell had a surprisingly good third party library for, called Pester) was out of the question. And: There was no version control! No wonder it got to 100k lines - most of the code was being entirely unused! But what?
The first few Saturdays I spent didn't go very far. I was mostly trying to figure out where the logic even began and ended in this mess of spaghetti. But eventually I started to piece together an overall structure for the thing. I made a pact to myself not to delete any of the many, many files that had accrued over the years until I knew every file that the code would touch first. I started doing some lightweight encapsulation with the -WhatIf flag, and with firebreaks (sticking `return` after each tiny function) so I could work on one linear piece at a time, and gradually tiptoe my way forward to the finish line. Eventually I got there, and managed to strip the whole thing down to about 1500 lines.
The joy of this kind of programming is that, even if you don't start with regression tests or modern practices, you do know from having ran through the original process a million times exactly what the end product is supposed to look like. Since no one is paying attention to your refactoring work in the first place, you're allowed to take your sweet time. To me it was obvious this mess of code was going to become a blocker for the business at the pace we were going: We were a SaaS, and yet we had to manually provision customers behind the scenes like this, in a process which took days to weeks? And we didn't even have a proper CLI we could call for this? Jackpot!
I left the company about 2 months after wrapping this project up and demoing it to my coworkers in search of better pay, but I'm really glad I got the chance to take such a fearsome codebase and tidy it up like that. There is just something satisfying about that kind of work, it's like gardening. I also got really, really good with Notepad++'s find and replace across all files functionality. There is in fact a skill to it! (Mostly that you should always, always find and replace the longest strings first!) Although I think these days I'd probably reach for a similar function in vanilla vim first, or even in bare `sed` if I was feeling gutsy.
(I'd end with "If you, dear reader, are in a similar PowerShell quagmire you need someone to pull you out of, I'm available for consulting at $X00/weekend", but alas. My weekends for the next few months are already sworn to another equally wonky project for my current employer. At least I'm getting overtime this time. ;) )
augustk 303 days ago [-]
> Mostly that you should always, always find and replace the longest strings first!
In Notepad++ you can mark "Match whole word only" or use a regular expression with word boundaries, e.g.
\bsome string\b
hiAndrewQuinn 303 days ago [-]
True and helpful, but perhaps not as universalizable as past me would have wished. I think IntelliJ or JetBrains or one of those big honking IDEs has support for proper intelligent refactoring of variable names like this.
windock 304 days ago [-]
I felt so much joy just reading this! Yeah, it is truly great to just simplify.
There are a bunch of reasons for that, but two interesting ones are (1) historical material is particularly welcome here, and (2) we want cohorts of new users to get an 'education' in the 'classics'. It's too easy for grizzled veterans to assume that everyone has already seen the perennials, but even they once read The Story of Mel, or whatever, for the first time!
stringtoint 303 days ago [-]
Nice post. In the past I've had colleagues (also senior people) who disliked touching "legacy" parts (e.g. Java based microservices compared to newer Express.js or Kotlin based microservices which don't provide half the value that the legacy parts do).
I also think that maintenance programming can be rewarding. However, I do believe that it does not make sense to throw entry-level engineers into the maintenance mess alone: they still need some guidance in my experience. Then you'll be able to reap the benefits such as getting better in debugging and understanding more about the business problems that the software is solving.
summerlight 303 days ago [-]
One more benefit is to learn how to tame complexity. The most complex piece of software doesn't come from nowhere; it is built over years, if not decades. People will never experience the same level of complexity in the "greenfield" programming. And that experience changes the entire mindset, from how to write the codes to how to structure the entire engineering org.
304 days ago [-]
Cthulhu_ 303 days ago [-]
I wish I spent a year doing this at my previous job instead of doing what I was hired to do, rebuild the project. The rebuild would've taken another 2.5 years, even if I had help.
But the existing project could've used some improvements; there was an inefficient SQL result to XML string to JSON thing happening with every request, so doubling response times could probably have been done within a week of work or so, and quadrupling it with a few weeks of work (there was thousands of LOC of SQL to XML code involved, but in most cases it could've been SQL to JSON directly).
Jabrov 303 days ago [-]
Let's say I wanted to do this. How do I actually get started?
Let's say I make a website that showcases my qualifications and lists out the services I offer.
I’ve only skimmed it and have started reading but the parts about highlighting/phrasing of one’s marketable skills seems good.
jakobson14 303 days ago [-]
After the NX bit, and cookies, and shadow stacks, and now tagged pointers (again) I'd rather stop pushing the guardrails for shitty code down into the hardware. It costs performance and the bandaging never ends.
302 days ago [-]
Rendered at 08:58:08 GMT+0000 (Coordinated Universal Time) with Vercel.
My experience is that this is almost never the case.
Generally, management does not understand or appreciate why it's a slog, why you might want to refactor this or that, etc.
Obviously a lot of this falls upon the engineer. It's your job to communicate and demonstrate. Helps if you have other engineers backing you up and presenting a unified message. Also helps if management is technical enough to understand what you're saying instead of simply believing -- or not believing -- you.
But maaaan. Uphill battle! I'm burnt out on it. I've decided I need to seek greenfield stuff. Maybe forever. But definitely for now.
Very few companies understand that the most valuable asset of a programmer is his mental model of your system. You need people who can do the hard work because they understand the system at an intuitive level.
Which is actually something a lot of management would prefer to not have. They often want people to deeply document a system, on the mistaken belief that this knowledge could be transferred out easily.
Unfortunately, it's not. Even a well documented system has inherent complexity that only working on the system would reveal. Otherwise, merely reading the source code would allow for a complete understanding and deskchecking of the whole system!
- Well documented, well maintained code that everybody could work on
- Quickly finished projects that nobody ever touches again, but only the programmer(s) have a mental model of it
The first option is more expensive at first, but way less expensive if you happen to have to rewrite it from scratch at some point (and if you are not maintaining and document it, that moment will come and it might not come at a point in time that fits the schedule).
The second option is less expensive first, but it will bite you in the arse if the programmer(s) are gone or forgot how it is done.
Everywhere else managers try to minimize risks, here they often just don't aeem to get the trade-offs involved.
https://www.theverge.com/tldr/2019/10/4/20899198/google-read...
I've got mixed feelings about this. Google's frequent "killings" have caused me to never trust them or want to buy in to their products.
But at the same time I think that it is good to "honor" "dead" products. Those products encompassed millions of hours of blood and sweat and toil and so on. I don't know the best way to memorialize that but sanitizing them from history entirely ain't it.
Also as you point out, getting funding for preventative maintenance or continuous improvement is extremely difficult in a corporate IT environment, especially if there isn't a product owner to advocate for the system and to secure funding. Although I do believe you need a product owner to drive the work to ensure it delivers business value, and not have it driven solely by developers.
May I ask what sort of company this was? Some "unicorn" startup? Stable oldcorp?
in mature industries, there aren't new revenue generating opportunities, there is only getting better at what you ordinarily do; all marketshare growth comes at the expense of competitors.
Refactoring it is just one of possible parts of the process, much opposed to 'yeah this legacy code is broken dogshit and needs to be rewritten even though the current software is already written, proven and works for 99% of the stuff we need' approach.
Clients and management generally understand that old code needs to be maintained. They don't love that reality, but that's not generally a tough sell unless they're completely clueless. And they're usually not completely clueless.
The tension in my experience comes when an engineer wants to perform something other than the fastest, sloppiest, filthiest duct-tape maintenance imaginable.
Management accepts the need for fixes, but often does not have the technical/domain knowledge to understand the short- and long-term merits of Proposed Fix A versus Proposed Fix B. And naturally, incentives heavily push management toward the quick fixes by default. Time is money, after all. And managers are expected to meet quarterly and annual OKRs. We don't have a good way to quantify the long-term value of a "proper" fix versus a "duct tape" fix. And so on.
So for me maintenance work is difficult to separate from notions of longer-term software maintenance such as refactors, dependency upgrades, etc.
I personally don't believe refactoring should ever be part of "maintenance".
Maintenance needs to avoid YAGNI and avoid the engineers natural desire for over-engineering and unnecessary perfect polishing and abstruse abstraction.
Refactoring is something that should be driven by new features (or worst case towards rewrite), and few have the skill to do it well.
I'm not arguing the sibling strawman that maintenance be "the fastest, sloppiest, filthiest duct-tape maintenance imaginable."
Excelling at maintenance is highly skillful work and often poorly rewarded.
Assuming we're talking about a program that does something useful and isn't just a screensaver, it's probably ingesting data from the outside world. That data can change in a variety of ways. API partners and users can introduce changes on their end and will eventually find new and annoying edge cases. Or maybe the volume of data increases by 100x and changes must be made on your end.
Or maybe your software sort of rots as the security vulnerabilities pile up. Eventually you've got to make major version or point upgrades to the framework and/or various dependencies.
We want to handle these demands judiciously and as a rule, yeah, refactoring isn't generally going to be the answer but sometimes it is.
Of course it shouldn’t, because software maintenance does not exist. Programs don’t degrade with use. All “maintenance” on software is, like new development, the implementation of new context->behavior combinations, and calling it something different is a result of applying an inappropriate physical-product metaphor.
The word "maintenance" doesn't imply some kind of physical product metaphor. Sorry. That's not what the word means.
There are several activities in software that are similar to maintenance of physical products. e.g.
* that parsing library that you used 2 years ago and was in good working order then, is now outdated and has known vulnerabilities that are not present in the latest version, apply update.
* Those cloud-hosted databases that you use are an older version that the cloud vendor has deprecated and will some time withdraw support for, apply migration to a later one.
CEO will start looking at this topic if app performance/stability metrics will start making impact on revenue stream, aka app is slow and stop working every day and customers go away because of that.
It's just a lack of understanding. They imagine they understand, because software doesn't get worn out, and doesnt degrade obviously!
Every larger company I've worked at had enough engineering overhead that they could afford to have small teams working on both reworking old stuff and creating new stuff, while the main teams kept the working stuff generating revenue.
As an embedded dev I have thousands of devices that were commissioned and are on 24/7 duty. The oldest still running fleets I believe date back to 2007. Some of them had firmware updates, but many of them (despite numerous bugfixes and improvements since) run on original software. Customers are reluctant to update if there's no specific need arises in their static operational environment.
Naturally in business logic world the requirements are never as static. But when they do change you can likely sell the management on it as necessary development.
I've found this to be true... to the point where some places even prefer manual maintenance to spending (potentially) a little more time on automation, even if that time would be paid back over the next couple of years.
I'm not adept at planning, selling my work, doing the manual maintenance, and doing the extra automation, as eventually that work becomes homework unless you get a boss that's willing to put in some effort here, which I haven't found yet.
But if the whole team is on board and has reviewed your proposal it's going to be more credible. And that's just good engineering anyway.
As the months and years wear on though, things change. Each bug has an obvious solution, and you’ll take the path of least disruption. Implementing new features is a matter of figuring out which trade offs you’re most ok with. Maybe you need to refactor something first, but that’s an obvious path itself.
You’re painted in a corner. A beautiful corner where the only safe thing to do is to tread carefully. But you don’t mind, because you know where you are and what you need to do to get to where you want to go.
Having done lots of both greenfield and maintenance work over four decades, I can unquestionably say that software maintenance is way, way more difficult.
I heard this first from Joel Spolsky, "it's easier to write a program than read it."
Which led me to write an article on the subject, "How To Read A Program," about the cognitive aspects of trying to learn unfamiliar code. https://www.itjungle.com/2009/01/28/fhg012809-story01/
The article uses AS400 applications as an example, but is general enough that it applies to applications in most languages.
This is a field begging for help from AI.
Most of the problems of API design could be remedied by people stepping through their own code and looking askance at the whole process.
If the latter is the case, then you get better at reading code by writing code. Writing lots of code puts those code-patterns into your long term memory, and then when you see those code-patterns again you'll recognize them.
For system design too - if you've designed lots of systems yourself, then when you see a new system, you'll be able to relate it to the systems that you've designed before.
So maybe building greenfield projects also makes you better at maintaining existing projects?
It'd be great if someone could point me to some existing literature on this topic. I've looked around and can't find any.
I think it does, because it builds a higher-level sense of how something "could" or "should" be and familiarity with thinking at the system level.
I've had a lot of problems with people who (seemingly) only have experience maintaining projects. They seem to have a tendency to focus narrowly on "fixing" a bug close to its source, and often lack understanding of a lot of fundamental technologies (because they only attend to things that have broken), and get stuck by a kind streetlight fallacy where they fix things in more familiar areas. The end result is ugly kludges and whack-a-mole bugs where a superficial "fix" in one area causes a new bug to pop up in another.
Only to an extent, i think.
> I've had a lot of problems with people who (seemingly) only have experience maintaining projects.
And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
I think "i've never bootstrapped a project before" is easier to cure than "I don't have a good sense of what's expensive vs what's cheap tech debt."
Some tech debt is basically 0% interest. Was this small hack bad? Yeah. Will it need to be fixed, eventually? Yes, for sure. But does it compound every day, or does it just sit there being a little yucky? Very easy to determine in hindsight. Very hard to predict as you write the hack. The end result being the simultaneity in my example. People will over-engineer things that won't end up being problems and under-engineer things that will turn out to require compounding hacks and it would've been cheaper to just get it right the first time.
> And similarly I've had problems with people who (seemingly) only have experience with starting a new project and then simultaneously over-engineering and piling on tech debt.
The best thing is to build a greenfield project and then maintain it for several years, fix your mistakes, then do the same thing on a new project again (including the maintenance).
You really need the full spectrum of experience, and put yourself in the shoes of the future maintainer when you're building something new.
They never have to maintain anything, some poor soul has to fix the broken software.
They also dont even know what mistakes they made, since they arent there to solve it.
I never felt like I had 1 year of experience 5 times because I moved between several kinds of projects at different lifecycle phases and with different constraints, and drew lots of parallels between them. At the end of five years I had project histories in my brain that spanned more than ten years. And got jobs that should have gone to someone with 8 years’ experience. I do not think this was a coincidence.
I maintain two major maps, which are what is the code logically trying to do, but also what exactly it's doing. (i.e. Sanitize inputs, check error conditions, do work, return result as a logical model and then "ensure foo is < 0 and > -10" etc. as the specific checks).
Let's hope what you write the first time is a good practice.
Some projects though just keep on trucking along and doing the next part with the newest shiny thing. You end up with five ways to access data, three ways to send emails and fifteen different methods of report generation. Frankenstein architecture! This architecture also emerges when you have leaders that don't make decisions and peeps debating all day on slack and then doing their own thing anyway because they didn't really want to offend anyone and everyone's ideas are great.
When I wrote that, in the wake of the 2008/9 meltdown, I intended to give programmers some career options to think about. More interesting and stable work out there than FAANGs and startups.
You're still somewhat biased, so you can get put on projects where the style is different from yours. If you happen to think that style is bad, it won't be fun to work on.
You shouldn't want to change the style. Chances are you'll do it incompletely (job change) or the time spent won't be worth it. If someone else comes along and doesn't like your style, we're just going to repeat this process.
There's been way too many different approaches that work for developers to be unified on what styles are best.
At my last job I had to work with a PowerShell codebase that `cloc`ed in at over 100,000 lines, and was absolutely ancient to boot, having been added to in bits and pieces by sysadmin after sysadmin over the course of 12 years total, in 3 different human variables-and-comments languages, all of whom were Windows power users with very clear ideas of what needed to be done, but also all of whom were very clearly learning to program for the first time, at least in the wonky PoSH paradigm. AKA: Classic big ball of mud situation. Every new customer had to have their details run through this codebase, however, and I saw the potential to eventually boil this down to a simple 6- or 7-parameter cmdlet (PoSH for "shell script"). I got my undersized chisel and toy hammer and got to work.
In some ways starting work on the proj felt more like beginning to dig into an ancient COBOL database in a System/360 mainframe than anything resembling the state of the art in 2023, or even 2011. There was a single VM that everything had to run on, and the script couldn't even be called remotely via SSH because the only way in was via a web browser based remoting session. The connection was spotty and would often drop halfway through coding in there in my dinky little Notepad++ session, requiring minutes of reauthentication just to get back to work. Unit testing (which PowerShell had a surprisingly good third party library for, called Pester) was out of the question. And: There was no version control! No wonder it got to 100k lines - most of the code was being entirely unused! But what?
The first few Saturdays I spent didn't go very far. I was mostly trying to figure out where the logic even began and ended in this mess of spaghetti. But eventually I started to piece together an overall structure for the thing. I made a pact to myself not to delete any of the many, many files that had accrued over the years until I knew every file that the code would touch first. I started doing some lightweight encapsulation with the -WhatIf flag, and with firebreaks (sticking `return` after each tiny function) so I could work on one linear piece at a time, and gradually tiptoe my way forward to the finish line. Eventually I got there, and managed to strip the whole thing down to about 1500 lines.
The joy of this kind of programming is that, even if you don't start with regression tests or modern practices, you do know from having ran through the original process a million times exactly what the end product is supposed to look like. Since no one is paying attention to your refactoring work in the first place, you're allowed to take your sweet time. To me it was obvious this mess of code was going to become a blocker for the business at the pace we were going: We were a SaaS, and yet we had to manually provision customers behind the scenes like this, in a process which took days to weeks? And we didn't even have a proper CLI we could call for this? Jackpot!
I left the company about 2 months after wrapping this project up and demoing it to my coworkers in search of better pay, but I'm really glad I got the chance to take such a fearsome codebase and tidy it up like that. There is just something satisfying about that kind of work, it's like gardening. I also got really, really good with Notepad++'s find and replace across all files functionality. There is in fact a skill to it! (Mostly that you should always, always find and replace the longest strings first!) Although I think these days I'd probably reach for a similar function in vanilla vim first, or even in bare `sed` if I was feeling gutsy.
(I'd end with "If you, dear reader, are in a similar PowerShell quagmire you need someone to pull you out of, I'm available for consulting at $X00/weekend", but alas. My weekends for the next few months are already sworn to another equally wonky project for my current employer. At least I'm getting overtime this time. ;) )
In Notepad++ you can mark "Match whole word only" or use a regular expression with word boundaries, e.g.
The joys of maintenance programming - https://news.ycombinator.com/item?id=2267261 - Feb 2011 (29 comments)
There are a bunch of reasons for that, but two interesting ones are (1) historical material is particularly welcome here, and (2) we want cohorts of new users to get an 'education' in the 'classics'. It's too easy for grizzled veterans to assume that everyone has already seen the perennials, but even they once read The Story of Mel, or whatever, for the first time!
But the existing project could've used some improvements; there was an inefficient SQL result to XML string to JSON thing happening with every request, so doubling response times could probably have been done within a week of work or so, and quadrupling it with a few weeks of work (there was thousands of LOC of SQL to XML code involved, but in most cases it could've been SQL to JSON directly).
Let's say I make a website that showcases my qualifications and lists out the services I offer.
How do I actually get a client?
I’ve only skimmed it and have started reading but the parts about highlighting/phrasing of one’s marketable skills seems good.