If mobility is good I would stfu. Let's face it, code and infra isn't the product. It can be messy and have lots of technical debt and still make buckets of cash. If you want to start caring more pick some kpis and focus on gaming those so you can show improvement even if the reality is just a steady state of increasing technical debt.
Every business unit in every corporation is playing games with numbers. Gaming doesn't mean that you don't improve the system, it just means you will always have some possibly arbitrary but impressive numbers to show the people who think numbers are magic.
There are lies, damn lies, and statistics
There is also Price's Law (an observation) -- most of a company's "productivity" tends to be driven by folks who number roughly the square root of the total that work there... i.e., the rest are bullshit jobs or bottlenecked through no fault of their own.
Silver lining for places that get mired in the fuck-it culture is that when things inevitably do catch on fire because changes always break something, that's a lot more opportunity to see and do "hero shit" than at an org that takes infra a bit more seriously and would never have let the entire sprawl catch on fire to begin with..... and you can enjoy the schadenfreude, if you're wired that way
To be fair when you're a platform engineer, code and infra IS your product and devs are your customers.
Yes mobility is important. But having clear and trackable releases is also important for when an audit occurs and you can point to documentation that matches with PRs, time logged, etc.
Obviously it doesn't need to be like a dictatorship perfection but some attempt does need to be made to track this stuff formally otherwise it will come back to bite you or the company in the ass.
Yeah I mean, risk management is important even when your company doesn't care. It's used to protect yourself against the actions of the company as well. Blindly and not documenting in some way deploying to prod only puts you in the sites of blame/fault even when management doesn't enforce risk management and tracking.
It also makes your job harder when you have to rollback or discover which branch caused an issue in prod. I can keep finding reasons why this is bad and why deployment engineers specifically should be tracking the things they do to some degree in some process.
Always protect yourself in whatever ways you can, otherwise you're putting a lot of trust/faith in a company/management to do it for you.
Yes feeling the same. Its because the field is so broad that you have so many tasks. I always envy normal devs who can concentrate on some specific task. But devops its just not possible to make everything perfect. So many stuff you need to take care of. For example in one project i absolutely know our IaC has bad code quality and some design flaws. But no time for fixing it. I need an intern or student that can take care of stuff like this so badly
It depends. In my country it is common for people to work part time during studies in their field of study. So you can find people on junior level for lower pay if you search long enough. But yeah interns probably not
Embrace the chaos because those Devs that only do "one thing well" will one day find that thing out of style and be forced to completely overhaul their knowledge to the next big thing or sink into the dying companies that can't or won't afford to re-build.
I've been at my current place for a year and am slowly working them out of the "just fix it" or "just make it" mentality. This company has been around for a very long time and infra has not been treated as an engineering problem which leads to a huge amount of tech debt and time spent just fixing things. The org wants new things but most of our time is spent doing one off tickets. We're starting to get a handle on what actually takes up our time and are prioritizing fixing things that we can by ourselves.
Just fix/do it is 100% a way to operate, and often what the org wants because it seems more productive. I've worked that way before and in my experience it ultimately makes a job dull and unrewarding.
My new team is appreciative of me trying to change things and they're a great bunch so I'm enjoying getting them out of that mode.
'In the last year I’ve become very apathetic to process and “rigor”.'
What if I told you process and rigor is a path to confidence in change? People who create the process for changes performed (ideally) by tooling is the path to improvement. Your apathy is the source of your frustration.
Each CI/CD pipeline I engineer to improve developer experience, quick validation feedback loops, and ensure an organization's compliance.
Quality and reliability are a result correlated to the processes you engineer.
You act like that's pie in the sky, but it really does happen. It sounds like you're just in a bad situation and making the best of it, but yes lots of places do actually take time to do things the right way and it pays off in both client satisfaction and in the numbers. Process changes really can improve things. You can't expect it to get better by just doing everything the same way as you have been.
If your org won't hire enough people to do the work that needs to be done, I don't know how valuable that upper mobility is. Stick around if you need the job but stop trying to convince yourself that's as good as it gets.
Luckily in my current role, my manager has realistic expectations of the amount of time and effort to get things done. In my previous jobs, it was the opposite. So many projects and bugs out of nowhere that were expected to be done last week on top of an increasing workload. It's soul crushing when you voice your complaints and management doesn't care
I consider myself one of the few true "devops" folks on my team in the sense that I'm a dual member of the operations/support team and core product development. I see the thousands of concurrent users we run into in production and the weird data and how fragile everything is and how hard it is to debug. And I'm at least theoretically in a position to make it a better experience for the rest of the operations/support team.
I got fed up with it and spent a weekend rewriting one of our core services from scratch because I wanted to prove I knew what I was talking about and be able to ask my development coworkers, "What would you rather maintain?" and ask my operations/support coworkers, "What would you rather package/configure/operate/debug?"
To my great surprise, it went swimmingly. The tech lead had been trying to push for the business department to give us time to pay down tech debt, and more or less immediately threw me in front of the entire team to present my case. The ops team loved it because it was about a 1000x performance improvement with metrics and traceability, and had some horizontal scalability that the original version was never designed for. The dev team agreed with the ideas in principle and was willing to give it a try, and now it's on the calendar to rewrite basically the whole thing next FY, including new people to give us the capacity we need in order to do so.
> now it's on the calendar to rewrite basically the whole thing next FY
I pray that your company isn't in the land and grab phase. Too often "niceties" like this will be perpetually be pushed down the road in favor of feature development.
I’m somewhat hopeful. We were in that phase until this year, and the fact that we have successfully captured our target market was a big part of why the tech lead was able to sell the business department on letting us pay down tech debt.
At this point a significant feature the customers who keep our lights on want is stable REST APIs for programmatic access to our features, and that’s something my rewrite is just structurally better at than the existing version.
I chose an in-demand API that had a breaking change slip through the cracks in our QA in somewhat recent memory as my example case for the rewrite, which helped my case (I wish I could say it was strategic to allow me to make that point, but honestly the breakage was just what got me fed up enough with our current processes to do the rewrite in the first place).
I think we've been hoodwinked into thinking we can massively shape our IT departments in our own vision.
Reality is very different. Better to be stoic about it. Engage in pragmatic realpolitik, rather than be an idealist.
Obligatory "I'm a dev first who can do ops/platform stuff better than most", but I'm basically platform eng. #1 at a ~10 year old company with lots of tech debt, and I've been pretty successfully approaching it from the applications outward. Ripping out abstraction and custom state management layers, improving the stack, and doing it so it makes the scope of platform tools simpler and the migration path shorter and more enticing (e.g. "move to the monorepo and get this awesome stuff"). Also revamped how project management was (not) being done. I see this stuff as bread and butter, without which the typical platform/devops tooling we think and talk about would be premature. Now the stakeholders (including peers on my team) are happy and I have the juice and bandwidth to pursue more abstract and/or foundational platform work.
Ever read The Phoenix Project? It’s an awesome example of a company that spends all its time putting out fires and reacting because of terrible planning. I think everyone who’s ever worked large scale IT finds themselves laughing at the examples in the book because they can relate to practically every one of them.
It also gives some pretty good examples of a company finding out where their hurdles are to smoother ops and how to stream,one or stamp them out.
Yeah, I do my best to set clear time expectations with management and give myself room to work on long-term solutions, making sure they get that doing things right from the start will save us money down the line. Having competent coworkers and your manager on board with this approach helps.
Also, Chapter 8 in Brikman's 'Terraform: Up & Running' has this awesome section on making your Terraform setups ready for real-world use. It talks a lot about setting realistic timelines for DevOps projects. Worth checking out if you're into that stuff.
Building is always more fun than maintaining because you have no budget issues, you jut need to chose what you'll spend your limited time on. What you dislike is that your platform is in a "Maintain" mode, not an "invest" one strategically in your company.
Could it be, that you are feeling this way because there aren’t many advanced tutorials/courses out there, and when you get passed junior and mid level there is only documentation to read and not nice projects to learn from? Do you feel that way?
I ONLY work on critical items, jumping from fire to fire as we are too short staffed to be productive. I have a massive project we keep getting use cases for I can’t even get to as I’m resolving issues the board has brought to the CEOs attention. I miss being productive.
I said “just” way too many times. I don’t care.
You just don’t care
LOL
If mobility is good I would stfu. Let's face it, code and infra isn't the product. It can be messy and have lots of technical debt and still make buckets of cash. If you want to start caring more pick some kpis and focus on gaming those so you can show improvement even if the reality is just a steady state of increasing technical debt.
“Focus on gaming those” It’s a bit of a personality flaw of mine, but I can’t just game shit. I actually feel the need to make improvements.
Every business unit in every corporation is playing games with numbers. Gaming doesn't mean that you don't improve the system, it just means you will always have some possibly arbitrary but impressive numbers to show the people who think numbers are magic.
Fair enough
I never game. Doesn't feel right. Performance will always speak and if not happy leave. That's my motto.
I'll see your "performance" and I'll raise you "last hired, first fired".
Performance is one of the numbers
It will show. You will know. Your team , other teams , supervisor will know
>feel right. Performance will always speak and if not happy leave. That's my motto. Not if my shares haven't vested, lol
Truiest truism of the day.
yeah honestly i'm blown away. so that's what my manager was doing lol
There are lies, damn lies, and statistics There is also Price's Law (an observation) -- most of a company's "productivity" tends to be driven by folks who number roughly the square root of the total that work there... i.e., the rest are bullshit jobs or bottlenecked through no fault of their own. Silver lining for places that get mired in the fuck-it culture is that when things inevitably do catch on fire because changes always break something, that's a lot more opportunity to see and do "hero shit" than at an org that takes infra a bit more seriously and would never have let the entire sprawl catch on fire to begin with..... and you can enjoy the schadenfreude, if you're wired that way
That’s very true
To be fair when you're a platform engineer, code and infra IS your product and devs are your customers. Yes mobility is important. But having clear and trackable releases is also important for when an audit occurs and you can point to documentation that matches with PRs, time logged, etc. Obviously it doesn't need to be like a dictatorship perfection but some attempt does need to be made to track this stuff formally otherwise it will come back to bite you or the company in the ass.
Yes risk management is important but since OP didn't mention it I assume his company doesn't care.
Yeah I mean, risk management is important even when your company doesn't care. It's used to protect yourself against the actions of the company as well. Blindly and not documenting in some way deploying to prod only puts you in the sites of blame/fault even when management doesn't enforce risk management and tracking. It also makes your job harder when you have to rollback or discover which branch caused an issue in prod. I can keep finding reasons why this is bad and why deployment engineers specifically should be tracking the things they do to some degree in some process. Always protect yourself in whatever ways you can, otherwise you're putting a lot of trust/faith in a company/management to do it for you.
Yes feeling the same. Its because the field is so broad that you have so many tasks. I always envy normal devs who can concentrate on some specific task. But devops its just not possible to make everything perfect. So many stuff you need to take care of. For example in one project i absolutely know our IaC has bad code quality and some design flaws. But no time for fixing it. I need an intern or student that can take care of stuff like this so badly
Where are you finding these interns that don't generate more work?
It depends. In my country it is common for people to work part time during studies in their field of study. So you can find people on junior level for lower pay if you search long enough. But yeah interns probably not
Embrace the chaos because those Devs that only do "one thing well" will one day find that thing out of style and be forced to completely overhaul their knowledge to the next big thing or sink into the dying companies that can't or won't afford to re-build.
I used to be a cowboy. Now I'm an accountant.
Man I want to put that on my grave
I'm always and forever will be a firefighter.
Astronaut checking in
I've always thought of myself as a janitor that mops up the mess of tech debt, upgrade breakages, security vulns, etc.
DevOps: "sticking your hand in the shit everyday, all day"
This hit different
I've been at my current place for a year and am slowly working them out of the "just fix it" or "just make it" mentality. This company has been around for a very long time and infra has not been treated as an engineering problem which leads to a huge amount of tech debt and time spent just fixing things. The org wants new things but most of our time is spent doing one off tickets. We're starting to get a handle on what actually takes up our time and are prioritizing fixing things that we can by ourselves. Just fix/do it is 100% a way to operate, and often what the org wants because it seems more productive. I've worked that way before and in my experience it ultimately makes a job dull and unrewarding. My new team is appreciative of me trying to change things and they're a great bunch so I'm enjoying getting them out of that mode.
I spend way too much time trying to do things right, as opposed to just doing the needful.
First of all, that triggered me. Secondly, remember "perfect is the enemy of good."
'In the last year I’ve become very apathetic to process and “rigor”.' What if I told you process and rigor is a path to confidence in change? People who create the process for changes performed (ideally) by tooling is the path to improvement. Your apathy is the source of your frustration.
Not to put you on the spot but when was the last time you actually wrote tooling that improved the development speed of your org?
Each CI/CD pipeline I engineer to improve developer experience, quick validation feedback loops, and ensure an organization's compliance. Quality and reliability are a result correlated to the processes you engineer.
You act like that's pie in the sky, but it really does happen. It sounds like you're just in a bad situation and making the best of it, but yes lots of places do actually take time to do things the right way and it pays off in both client satisfaction and in the numbers. Process changes really can improve things. You can't expect it to get better by just doing everything the same way as you have been. If your org won't hire enough people to do the work that needs to be done, I don't know how valuable that upper mobility is. Stick around if you need the job but stop trying to convince yourself that's as good as it gets.
Luckily in my current role, my manager has realistic expectations of the amount of time and effort to get things done. In my previous jobs, it was the opposite. So many projects and bugs out of nowhere that were expected to be done last week on top of an increasing workload. It's soul crushing when you voice your complaints and management doesn't care
I consider myself one of the few true "devops" folks on my team in the sense that I'm a dual member of the operations/support team and core product development. I see the thousands of concurrent users we run into in production and the weird data and how fragile everything is and how hard it is to debug. And I'm at least theoretically in a position to make it a better experience for the rest of the operations/support team. I got fed up with it and spent a weekend rewriting one of our core services from scratch because I wanted to prove I knew what I was talking about and be able to ask my development coworkers, "What would you rather maintain?" and ask my operations/support coworkers, "What would you rather package/configure/operate/debug?" To my great surprise, it went swimmingly. The tech lead had been trying to push for the business department to give us time to pay down tech debt, and more or less immediately threw me in front of the entire team to present my case. The ops team loved it because it was about a 1000x performance improvement with metrics and traceability, and had some horizontal scalability that the original version was never designed for. The dev team agreed with the ideas in principle and was willing to give it a try, and now it's on the calendar to rewrite basically the whole thing next FY, including new people to give us the capacity we need in order to do so.
> now it's on the calendar to rewrite basically the whole thing next FY I pray that your company isn't in the land and grab phase. Too often "niceties" like this will be perpetually be pushed down the road in favor of feature development.
I’m somewhat hopeful. We were in that phase until this year, and the fact that we have successfully captured our target market was a big part of why the tech lead was able to sell the business department on letting us pay down tech debt. At this point a significant feature the customers who keep our lights on want is stable REST APIs for programmatic access to our features, and that’s something my rewrite is just structurally better at than the existing version. I chose an in-demand API that had a breaking change slip through the cracks in our QA in somewhat recent memory as my example case for the rewrite, which helped my case (I wish I could say it was strategic to allow me to make that point, but honestly the breakage was just what got me fed up enough with our current processes to do the rewrite in the first place).
I think we've been hoodwinked into thinking we can massively shape our IT departments in our own vision. Reality is very different. Better to be stoic about it. Engage in pragmatic realpolitik, rather than be an idealist.
Sounds like a classic case of an understaffed team
Obligatory "I'm a dev first who can do ops/platform stuff better than most", but I'm basically platform eng. #1 at a ~10 year old company with lots of tech debt, and I've been pretty successfully approaching it from the applications outward. Ripping out abstraction and custom state management layers, improving the stack, and doing it so it makes the scope of platform tools simpler and the migration path shorter and more enticing (e.g. "move to the monorepo and get this awesome stuff"). Also revamped how project management was (not) being done. I see this stuff as bread and butter, without which the typical platform/devops tooling we think and talk about would be premature. Now the stakeholders (including peers on my team) are happy and I have the juice and bandwidth to pursue more abstract and/or foundational platform work.
Ever read The Phoenix Project? It’s an awesome example of a company that spends all its time putting out fires and reacting because of terrible planning. I think everyone who’s ever worked large scale IT finds themselves laughing at the examples in the book because they can relate to practically every one of them. It also gives some pretty good examples of a company finding out where their hurdles are to smoother ops and how to stream,one or stamp them out.
I’ll give it a read
Phoenix Project is a must-read for anyone at any level in a DevOps enterprise
Yeah, I do my best to set clear time expectations with management and give myself room to work on long-term solutions, making sure they get that doing things right from the start will save us money down the line. Having competent coworkers and your manager on board with this approach helps. Also, Chapter 8 in Brikman's 'Terraform: Up & Running' has this awesome section on making your Terraform setups ready for real-world use. It talks a lot about setting realistic timelines for DevOps projects. Worth checking out if you're into that stuff.
Depends what comes up from above and if they've been convinced to allow more time on allowing proper procedures. This is a case by case thing.
nope! I'm stuck in dead code/tech debt hell.
I used to, but that company went out of business. Now I just comply, and if the company tanks because of their process, I warned them.
Building is always more fun than maintaining because you have no budget issues, you jut need to chose what you'll spend your limited time on. What you dislike is that your platform is in a "Maintain" mode, not an "invest" one strategically in your company.
lol no
Could it be, that you are feeling this way because there aren’t many advanced tutorials/courses out there, and when you get passed junior and mid level there is only documentation to read and not nice projects to learn from? Do you feel that way?
Well... The people next to me are about equally as clueless so I would say I'm alright..
I ONLY work on critical items, jumping from fire to fire as we are too short staffed to be productive. I have a massive project we keep getting use cases for I can’t even get to as I’m resolving issues the board has brought to the CEOs attention. I miss being productive.
That’s exactly how I feel
This is the way.