Discussion
Loading...

Post

  • About
  • Code of conduct
  • Privacy
  • Users
  • Instances
  • About Bonfire
Stefano Marinelli
@stefano@mastodon.bsd.cafe  ·  activity timestamp 3 days ago

About a year ago, a client I've worked with for over fifteen years informed me that some of their "less critical" servers would be migrated to $CLOUDPROVIDER. According to them, this provider would guarantee an efficient management panel, "more freedom for their devs", and lower costs. This didn't impact me financially but, on an ethical and personal level, I warned him about the potential problems. Yet they decided to move forward, aided by the arrival of $YOUNGDEV who "has worked with it, it's reliable, and everything works fine". Again, I warned them (where are the backups? A disaster recovery plan? etc.) but they insisted: $CLOUDPROVIDER is efficient and gives us everything.

I studied their plan and immediately understood that their "cost-cutting" strategy wouldn't work: I know their workloads, and the plan they chose was insufficient. Needless to say, a few days later they went down and had to make an "emergency" purchase of the next tier up. The cost? Higher than their previous server infrastructure.

I heard nothing more about these workloads for almost a year but my monitoring tools still were marking them down, from time to time. Then, I get a phone call this afternoon. $YOUNGDEV asks me for support. He doesn't explain, but I immediately understand it's one of those workloads. A serious problem, and they don't have a backup of the database. They don't have a test environment to run diagnostics. The DB is very large, and they don't know what to do. My predictions - not even my worst ones - had come true.

I was running between two appointments. I only remarked that this situation could have been avoided and that it's not something I manage or can manage, but I nonetheless suggested we sync up tomorrow morning. I'm not going to get my hands dirty, but still, $YOUNGDEV is in trouble, and I offered to take a look to suggest a strategy. I then asked for the access credentials to $CLOUDPROVIDER, considering that up until a year ago, I managed all of these workloads. He replied that he "doesn't know if he can give them to me" and that he "would have to ask his bosses". I pointed out that if he wants my help, I need something - I don't even know how $CLOUDPROVIDER grants access to data (or if it does) - how can I give him advice?

It's 18:30 and I have received nothing. Tomorrow morning, if the phone rings, I will answer, but at this point, I won't do anything. I prefer, albeit reluctantly, to completely end the relationship with this client.

If this is the price of dignity and respect, I'll gladly pay it.

#IT #SysAdmin #HorrorStories #OwnYourData

  • Copy link
  • Flag this post
  • Block
Chuck
@cheeldcharles@mastodon.social replied  ·  activity timestamp 6 hours ago

@stefano I'm getting ready for Halloween, but don't let them do it. Everything is going to the cloud (where we lose control of our own stuff) and AI (so they can learn how our mind motivations and impulses work-for some reason). I will get more into it later, but wantedto make a place mark before I get Hallowed Eved tonight.

  • Copy link
  • Flag this comment
  • Block
Matthias Mair
@matmair@mastodon.social replied  ·  activity timestamp 3 days ago

@stefano I am on the other side of something like this rn. IT middle mgmt in a architecture role and up top direction is cloud-first / cloud-only. Really hard to help steer a ship in such a constellation, especially with the somewhat dooming financial downturn from trade uncertainties.

I think “the cloud” has its value for highly elastic / scaling workloads. Stable workloads should be at least hybrid if not fully onPrem in some Colo - Datacenter operations is not some secret magic.

  • Copy link
  • Flag this comment
  • Block
subnetspider
@subnetspider@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@stefano It's insane how higher ups complete ignore all potential risks when the cloud is involved. Once they hear "potential savings", it's all over.

But I see the same ignorance every day, where businesses prioritize saving pennies while risking tens of thousands in losses.

  • Copy link
  • Flag this comment
  • Block
subnetspider
@subnetspider@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@stefano In my case, I always recommend that they get a second internet connection, as the clients I work with run their workloads on-prem. However, rather than investing a few hundred euros in a 4G/5G router every ~5-10 years and paying around 10 euros per month for a SIM card, they risk losing thousands in sales every day when they're offline.

I just can't understand how a business owner could justify taking such a huge risk. It's essentially saving pennies on the dollar. Yet when their DSL or cable does go down, they demand immediate support and somehow blame us for their losses. That's basically like living without health and car insurance.

  • Copy link
  • Flag this comment
  • Block
DeManiak 🇿🇦
@kaasbaas@mastodon.africa replied  ·  activity timestamp 3 days ago

@stefano I think you have seen enough to know this, but for somebody with less experience, hear this: GET IT IN WRITING.
Whatever it is,IN WRITING.

if you got it verbally, reply with email, mentioning what they said.
Make them reply with a confirmation/acknowledgement.

  • Copy link
  • Flag this comment
  • Block
gunstick
@gunstick@mastodon.opencloud.lu replied  ·  activity timestamp 3 days ago

@stefano if you sent them an email a year ago, warning about their cloud setup not being adequate: take that email, paste it with timestamps and all into the reply, and send it off as is.

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@gunstick yes, I have some e-mails - and everything that is happening now is in those e-mails.

  • Copy link
  • Flag this comment
  • Block
Inarticulate Otter
@InarticulateOtter@mastodon.social replied  ·  activity timestamp 3 days ago

@stefano following to see outcome.

Historical unrelated anecdote, small render farm, storage/render-engines/edit-suites all connected by multiple fibre cards. We were hitting bandwidth constraints moving data 20 METRES.

The cloud sales people convinced management that storage/render could magically be done off site. The national infrastructure at that time simply had insufficient bandwidth, not a matter of opinion. I left, I think they limped on with a foot in both camps.

  • Copy link
  • Flag this comment
  • Block
Phil Sherry
@philsherry@mstdn.social replied  ·  activity timestamp 3 days ago

@stefano I’m guessing $YOUNGDEV is a Full-Stack Overflow Developer and nobody told them about database backups.

You should thoroughly wash your hands clean of any involvement there and have an easier life for it.

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@philsherry I totally agree.

  • Copy link
  • Flag this comment
  • Block
JimmyChezPants 🇨🇦
@jpaskaruk@growers.social replied  ·  activity timestamp 3 days ago

@stefano this is more or less how I lost the best job I ever had.

  • Copy link
  • Flag this comment
  • Block
scrottie (he/him/them)
@scrottie@bsd.network replied  ·  activity timestamp 3 days ago

@stefano $client is doing backups across datacenters. They had three servers in colo, with good redundancy (except for the colo itself, which did lose power once) costing a few hundred a month. Got bought and moved to the cloud, incompletely, before money ran out. Now half straddling the colo and a well known cloud provider. Cloud provider is charging them the price of six top-tier devs, every month.

  • Copy link
  • Flag this comment
  • Block
scrottie (he/him/them)
@scrottie@bsd.network replied  ·  activity timestamp 3 days ago

@stefano They keep having meetings and discussions about ways to cut capacity to reduce cost. Tongue in check, a few times now, I've pointed out that they still have all of the high end servers sitting in colo costing basically nothing that would make fine storage servers for *backups or fail-over servers. I'll take pregnant pause for $500, Alex.

  • Copy link
  • Flag this comment
  • Block
sysop
@sysop@runbsd.duckdns.org replied  ·  activity timestamp 3 days ago

@stefano But... the moral higher ground, could be helpin' them out. Not for free though.

  • Copy link
  • Flag this comment
  • Block
sysop
@sysop@runbsd.duckdns.org replied  ·  activity timestamp 3 days ago

@stefano wow. Whatever decision/route that you'll take, it'll be correct, even if ... seems wrong. I know the feeling though.

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@sysop I'm quite curious to see what's going to happen, tomorrow.

  • Copy link
  • Flag this comment
  • Block
Chuck
@ChuckMcManis@chaos.social replied  ·  activity timestamp 3 days ago

@stefano It is the price of integrity. You could always give the client a price sheet for returning the services on prem at their previous level of reliability. They would probably pay for it by not having to pay for $YOUNGDEV any more.

I was always surprised when the biz types were amazed that I knew more about their business and costs than they did. I have a hard time relating to people who just learn surface details and act on those.

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@ChuckMcManis What has always surprised me about some companies is that their eagerness to do something "new" or "different" drives them to destroy what made them prosper for years, in pursuit of what is clearly a utopia upon even a basic analysis. But I've gotten used to it by now.

  • Copy link
  • Flag this comment
  • Block
Recovered Expert
@RecoveredExpert@mastodon.social replied  ·  activity timestamp yesterday

@stefano @ChuckMcManis are you sure it’s the eagerness to do something new? Cases I’ve seen it’s almost always the vague(!) idea of „it’ll be cheaper“ or „it’ll be better“, often based on claims or even promises made by a new employee, contractor or consultant. When the people responsible at the company feel, are old to be or actually are „behind“ on what’s new or en vogue in terms of technology they’re particularly vulnerable to such exploits.

  • Copy link
  • Flag this comment
  • Block
oxy
@oxyhyxo@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@stefano @ChuckMcManis the well worn path has no surprises, including no surprising extra income. New and shiny offers the possibility that more money may magically appear from some new as yet unimagined source

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@oxyhyxo @ChuckMcManis ...or it can destroy all your business in a second.
The service they're trying to fix is a service they're providing to a client - one of their oldest clients. And not the most patient ones.

  • Copy link
  • Flag this comment
  • Block
oxy
@oxyhyxo@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@stefano @ChuckMcManis yes but thinking about magic beans is more fun. Reality is depressing 😢

  • Copy link
  • Flag this comment
  • Block
fedops 💙💛
@fedops@fosstodon.org replied  ·  activity timestamp 3 days ago

@stefano this!

Also it seems that a lot of those same people have no grasp of their actual operating costs of on-prem equipment. I have reviewed the IaaS cost sheets for the workloads they want to transfer, and when remarking this is app. 4x what our on-prem cost is the only thing I got is a bewildered stare. Sometimes followed by "but we don't need the people anymore". Sure, VMs in aws or azure administer themselves and backups just happen.

Complete lala land.

@ChuckMcManis

  • Copy link
  • Flag this comment
  • Block
Stefano Marinelli
@stefano@mastodon.bsd.cafe replied  ·  activity timestamp 3 days ago

@fedops @ChuckMcManis I agree. And the scariest thing is: when they go "to the cloud (TM)", they just stop caring about backups. And when I ask "where's a backup of your data", they're usually puzzled: they think that in the cloud, you don't need backups.

  • Copy link
  • Flag this comment
  • Block
fedops 💙💛
@fedops@fosstodon.org replied  ·  activity timestamp 3 days ago

@stefano much like with "ai" a large part of the current crop of IT manglement absolutely does not understand what "cloud" even means. Or proper IT services outside of the ITIL BS for that matter.

To quote my favorite, Admin Zen: "Nobody wants backups, everybody wants restores. To get restores you need backups."
@ChuckMcManis

  • Copy link
  • Flag this comment
  • Block
fedops 💙💛
@fedops@fosstodon.org replied  ·  activity timestamp 3 days ago

@stefano @ChuckMcManis for reference: https://adminzen.org/

  • Copy link
  • Flag this comment
  • Block
Chuck
@ChuckMcManis@chaos.social replied  ·  activity timestamp 3 days ago

@stefano I've found this is often driven by a manager's desire to 'have an impact' which, for them, means "improves the business" not "keep it going." Their analysis on everyone I've dug into looked entirely at the possible benefits and *always* minimized and risks or downsides. For new managers especially "do what the old manager did" is not a recipe for moving up the ladder.
1/2

  • Copy link
  • Flag this comment
  • Block
Chuck
@ChuckMcManis@chaos.social replied  ·  activity timestamp 3 days ago

@stefano I have headed off one of these by having a relationship with both the manager and their manager that allowed me to talk to both without them feeling threatened. But too often if you try to argue a manager out of their plan they will assume you're trying to sabotage their career (they are never thinking about the business it seems). So it doesn't help.
2/2

  • Copy link
  • Flag this comment
  • Block
Nick
@nick@shore.me.uk replied  ·  activity timestamp 3 days ago
@stefano@bsd.cafe @ChuckMcManis@chaos.social they also seem to never understand the cost of failure, and don't have the right plans in place
  • Copy link
  • Flag this comment
  • Block
Log in

bonfire.cafe

A space for Bonfire maintainers and contributors to communicate

bonfire.cafe: About · Code of conduct · Privacy · Users · Instances
Bonfire social · 1.0.0-rc.3.21 no JS en
Automatic federation enabled
  • Explore
  • About
  • Members
  • Code of Conduct
Home
Login