Adventures In Automation

Adventures in Automation

June 6, 2023 | Compiler Team

Automation and management

Compiler • • Adventures In Automation | Compiler

Adventures In Automation | Compiler

About the episode

Repetitive tasks can be the worst part of a job. They take time, they take focus, but they don’t typically spark creativity. That is, until you try to automate them.

What kinds of tasks are ripe for automation? Which ones are best left to do yourself? And what does it take to automate away the drudgery?

Compiler team Red Hat original show

Subscribe here:

Transcript

It is almost compulsory, not even a necessity, because when you start working, that's my work experience. When you start just trying to solve a problem, you realize that some things tend to happen, not once, but twice, thrice, and then it becomes... not a daily occurrence, but it's something that happens almost all the time and keeps happening on the intervals and stuff like that. So then you realize that you cannot keep doing things by hand because then you have the problem that is error-prone, and second becomes boring. And one thing that happens to people when get bored is that they make mistakes. And me in particular, I'm really good at making mistakes when something is actually boring me to death. So one of the things that I try to do is just to automate it if it's possible, and also to document or make it easy to use.

That's Jose Vincente[sic] Nunez. And if he sounds familiar, it's because he was a guest on our episode about sysadmins. He shared something with us in that episode that stuck with me. Sometimes when trying to automate a repetitive task in order to save time, you end up spending way more time than you expected to. I thought that was pretty interesting and I wanted to dig into that concept a little bit more. When does it make sense to automate? When should you avoid it? And most importantly, what can you gain from automation?

This is Compiler, an original podcast from Red Hat.

We're your hosts.

I'm Brent Simoneaux.

And I'm Angela Andrews.

We go beyond the buzzwords and the jargon to simplify tech topics.

Today, we hear about adventures in automation.

In case you missed our episode on sysadmins, Jose Vincente[sic] Nunez is a DevOps engineer for a FinTech company. He has years of experience in the field, and automation is a big part of his job. Like he mentioned earlier, automation is something to be embraced, especially with repetitive tasks. It saves time, provides consistency, and it preserves your sanity. But even though it's something he does on a regular basis for systems he works with every day, he can still find himself lost in the puzzle of automation.

In my head, I thought that I understood how it worked. Then you start testing and playing with that, and then you start finding a small box or unexpected behavior, and then you have to spend time trying to figure out why it's not working the way you expected. By the time you realize that or you figure out what was wrong, you spent X amount of time just testing or debugging something that was supposed to be straightforward.

That is so true.

Yeah.

It is a rabbit hole. Okay? When you start trying to automate something, again, that you think it'll be, "Oh, this is simple and we'll figure this out." The time you spent rerunning and reapplying the automation and then rolling it back and seeing if that worked and watching the error message, it's a whole thing. It was supposed to be straightforward, but...

It's like opening up a time warp where you just like-

Where'd the day go? I feel him on that one.

And I assume it could be frustrating too because if it's something that you do over and over again, you know the process and so you think, "Oh, I can just write a script or write a module or whatever, I'll just make sure all the commands are the same." And then you run it and it's not what you were expecting. Something goes wrong and you're like, "What is going on?"

I think this is something we can all relate to, right Johan?

Yeah. I don't do all that much automation, but in the past I've focused on an interesting problem and lost track of time. But if time saving were the only benefit for automation, then it can sometimes be a dubious proposition.

What time? What time are you saving, exactly?

Who needs to keep track of time? But when you have a process that you need to roll out for multiple machines, the idea of writing a script becomes much more attractive. Jose told me why it's important to take your time, even when automating something, like patch updates for the switches on his company's network, because a mistake could mean costly downtime.

So it is a combination, like say that I'm just fixing one switch with a handful of servers connected and they're used for test or QA, is an acceptable risk. You are going to modify or update 500 switches where you have a bunch of servers connected to them. Then actually we're talking about they cost the same, but the consequences are dire if actually you have half of those 500 switches failing just because the patch wasn't good. Same for hardware components. You're fixing probably one GPU card on a machine? Okay, yes, they're expensive, but maybe you can afford to be down while you roll back the change or figure out what went wrong with the update. If you're talking about maybe an FPGA card that costs like $20,000 and the server is used for low latency trading, then that's where you start thinking, "You know what? Maybe it's not such a good idea."

Oh, I'm sorry. This is funny to hear because we've all been there, we're doing our best. We're testing it on the smaller QA test systems and we're thinking, "Oh, okay, this works. This looks good. Everything passed all the tests." But then you went and deployed it to a thousand systems and you know, just watch those lights light up on the monitoring board of all these systems going down off of the network and you're just sitting here going like, "Oh my God."

Oh no!

Where's the undo button?

Right.

So what does it mean to take your time, right? I mean, we talked about testing your scripts, adjusting them, and then testing them again and again, until you're confident beyond a reasonable doubt that the update's going to roll out the way that you want it to, that it's going to be smooth. That way you're not going to brick a large chunk of your hardware and you're going to keep your infrastructure running. And for that really expensive piece of hardware, that FPGA card that's like $20,000? Sometimes it's okay to do things by hand. Now, another way to minimize things going wrong is to keep the scripts simple-

Okay.

And have them do one single job. And that's not always as easy as it seems because it can be tempting to add functionality to a script, and there can also be pressure from other teams to add functions as well. Now, over the years and over his career, Jose has had to learn an important skill.

I learned how to say no. So it's like, "Can you add this feature to put this data into the database?" "No." "Can you add this feature to launch a docker container and a-?" "No. What we can do, we can do this." But you don't need to change the tool. Keep the tool simple and anything that you want to put on top of that, we can do it.

All right, let's all say it together. No.

No!

No.

Okay. No, it's a complete sentence. It gets your point across and it is okay to lead with no, because not every request that you get requires your attention or not your immediate attention. So it is okay to say no, learn how to say no and be comfortable in it, because it may save you in the long run. Because you can always come back and say yes. It's hard to come back from saying yes to a no. So think about that.

Yeah, if you start with that simple script and you start adding things to it's harder to remove those features than it is to add them later on.

Yeah.

That's a great point.

So we're talking about automation and there's this famous automation instructor, and one of the words, this phrase he always says is, "Keep it simple. Complexity kills productivity." And I believe that wholeheartedly, you want to do things in modular, small, little piecemeal things. You don't want to go messing up a whole lot at once because it's really harder to undo something that is complex and it's really harder for someone else to come behind you and understand what you did when it's so complex. So yeah, I am all for putting the brakes on.

Well, it sounds like there's a lot of benefit to simplicity here, right?

For sure.

I think that there, it's not because you don't want to... It's not that you're lazy and don't want to do more work, right? It's because it has all these really great effects.

And keeping it simple makes it easier for the thing to just work and just keep working and you're minimizing the surface area for problems to occur either now or later on. Saying no is really great for keeping it simple. And those simple scripts, well, they can last a lot longer than you might imagine.

You'd be surprised. I seen pieces of code that are 20 years old and they're still kicking. Perl especially, oh my God, these guys, I'm telling you, if there is a catastrophe, like an asteroid hits the Earth or something like that, you will see the roaches and the Perl scripts taking over. Probably those will be the only two things that will survive because they're so resilient.

That was a perfect visual. Perl will never die. Neither will cockroaches, that has already been established.

Yeah.

How funny is that?

That's great.

I love it.

I love talking to Jose.

So what is it about these scripts that keeps them around for so long?

Well, part of it is because not a lot of people know Perl anymore, and if they work, you don't touch them. They're doing whatever task they were designed to do, and they do them very well. That's the real reason. No one knows Perl anymore. It's such an old scripting language, but it's very solid, whatever it's been tasked to do, oh, it does it and does it well. Seriously. So that's my one big reason.

Yeah, that's essentially what he told me, is that he said he was looking back over some old code and he found some few scripts that were still kicking around in production that he wrote years ago, and they still worked because they do the one thing, that you mentioned that they should be doing, they're just focused on a singular thing.

They do one thing.

They do one thing. They're very simple. They don't do that one thing in a complex way, and no one is allowed to touch them or add anything to them or change them in any way. So because software doesn't really decay, if a thing works and it's a thing that you need to have running for years and years, just don't touch it, unless you absolutely have to, and it might outlive you.

Might outlive us all.

Yeah. Except for the cockroaches.

Except for the cockroaches.

Eventually they'll become sentient and take over those Perl scripts and change them for their own needs.

And we won't be here to see it. So that sucks.

No, we won't. No.

Yeah, I don't think I want to be here to see that.

Yeah. So next we spoke to Patrick Arminio. He's a developer advocate at Polygraph QL. And for him, automation is about leaning into a self applied stereotype about developers.

I guess I'm a bit lazy in terms of, I guess most of programmers are lazy. So I really... Well, other than being lazy, I'm also, I tend to procrastinate a lot, and especially tiny tasks because I say I can do those anytime. And I realized that after a while, especially for some of the automation of, I guessing we talk about in Strawberry, there were a few things I always ended up putting off weeks or even months at a time sometimes, and I noticed that and said, "I'm going to automate this so I don't have to think about them too much."

Now, Jose said something similar, about programmers being lazy, but he kind of caught himself and rephrased it in a way that I think is much more accurate. "It's not necessarily about laziness, it's about being efficient with your time." Right? If you can automate a thing, something that you do on a repetitive basis, you can avoid spending all that time doing the thing over and over again and spend that time focusing on something else instead. And writing that script can be an interesting challenge in itself.

It's definitely more fun. I mean, seeing something that works without having you to do anything, it's very nice. I mean, even if it takes more time to actually automate it, then actually do the task. I think it's very fun. And sometimes it's actually a good way to learn new skills because you can say, "Oh, I'm going to try this new library, or I'm going to try this new language."

He's right about having that moment where you've automated something that is so cool and it just makes things so much easier for you. It is an amazing feeling, and when it works, it's an amazing feeling, and that's just one less thing you have to do. So you can go off and learn new things and try new things, but this one thing that may have plagued you or been in your back pocket and you kind of put it off and put it off, once you've finally done it and got all the bugs out and it works like it was supposed to work. That's pretty clutch. I've had that feeling. And it is a really good feeling.

On top of that, you get to learn something new as well.

Yeah.

And how often do we make that point on this show? That it's super important to just keep learning things in this industry. This is one way to do that while also being very productive, even if the time spent doesn't initially look like a good trade-off. You spent a lot of time automating and you're not sure if you're saving all that much time on the backend, but you're learning something new. You're getting something that's consistent and you are putting in that investment for time saved later on. So let's hear a little bit about what Patrick was being allegedly lazy about. He's a maintainer for an open source project that's pretty popular, and that means people are submitting pull requests. That also means that there's some regular work for the release process once those pull requests get merged.

I think the fact that wasn't automated, I would tend to postpone it a lot of times, which also means I would forget how it works. So I would need to read the docs. Because it's something... I mean, releasing a package is not something I used to do often in the past. So I would need to check what are the commands I need to run? What do I need to check to make sure that everything works?

So it's a process that wouldn't take a huge amount of time, a little more than 10 minutes each time and a few times a week. But he had to often go back and relearn how to do it, which would probably increase the amount of time every time. And not getting to it would cause delays, not just for him, but for his whole team. He was a bottleneck and that didn't feel right. So he decided he'd had enough and he started to automate the process. And now with the automation, it's much smoother. And if he's unable to do it himself or if he wants to take a break, someone else can step in and do it.

Oh, that seems pretty great, right?

Yeah, it does. That's how it started anyways-

Okay.

... but maybe Patrick and Jose need to have a little bit of a chat about scope creep.

The initial scope was to automate the release of this library, and then we added the invitation, we added the tweets, release cards, and adding a message to basically compile the form to get a sticker. So we definitely went... Well, I definitely went overboard and added new features because I don't know, it's fun to automate things and once you start, then it's hard to stop. I think the only thing that kept me from adding more features is just the fact that at the moment, it's like a lot of scripts, GitHub actions, and I think it's a bit hard to maintain. It's also not easy to test because you have to push on GitHub and wait for everything to run to test it.

So he's saying that he started wanting to automate the release of a library, and then he was just like, "Oh, if I can automate that, I can also add in a tweet about it." Right?

Exactly.

"I can also add in a release card. I can add in a message to so-and-so." Right?

Yeah. He wanted to add basically little, some of the little ancillary stuff that they do to build the community, right?

Yeah.

They wanted to announce that there's a new release that's available, so that people, users can know that they can download something new. They want us to add a little message for the person who wrote the pull request as a little thank you to say, "Hey, your code has made it into the main branch and we've released it, so thank you very much." There's a little invitation to a Discord server to say, "Hey, if you want to chat with us, you can become part of the community that way," and he just kept adding another thing, one after the other, and it seems like it's gotten a little bit too big to handle.

Yeah, it definitely sounds like Patrick and Jose need to have a talk, have a little chat.

It started out simple, and then it's not so simple anymore. Patrick wrote up a pretty successful script, even though it's got all those fancy features, it's been fairly successful and fairly smooth, and it's so successful actually, that it's making its way into other open source projects whose maintainers were impressed by the script that Patrick wrote. Now, Patrick is keen on automating a lot of things. It's fun. It keeps him from being a bottleneck, but he does think that automation should have its limits.

Definitely the human interaction is something I'm afraid of losing. I really like being able to, I feel like I've said this a lot of times. I've love to people to be welcome when I do a request. I don't want to just approve it. I want to approve with a message saying, "Oh, this is nice."

There's been a push on his team to automate a lot more than what he thinks is necessary, and he's worried about having scripts take over what really helps build open source communities, even if it does mean doing some of that work himself.

There's something to be said for being able to interact with end users and having those conversations and being able to help them while you're helping yourself and your own education. So I feel him on that. There's sometimes where some things don't need to be automated, especially when there's a little bit of human interaction that can help. As you know the more remote we are, the more we're losing those types of interaction. So I don't blame him for wanting to keep a little bit of that. It's probably what makes his job very rewarding.

There's something to a very personalized, unique message to someone, not just an automated response that you-

Thank you, first name and last name.

Yeah, exactly. Yeah. The form fill. Yeah.

Well, there's something to receiving that too, to getting that little message to know that you're appreciated and you can tell when it's an automated message and when it's been something that's actually been handwritten for you-

Yeah.

... maybe not handwritten, but custom-

Personalized, Johan is sending it through a carrier pigeon, like a-

Calligraphy.

... handwritten note. Yeah.

So those are Patrick's projects, and again, he's really just been getting started with automation. Our next story is from someone who's working on a massive enterprise scale automation project. This one has the same benefits as the others, but instead of saving a little bit of time and providing a little bit more consistency, it's designed to smooth out a complicated process that's a fundamental part of a financial institution's business. One that was extremely prone to error. Satyajit Lele is a senior consultant here at Red Hat, and he's been working on a massive automation project. They've had a process cobbled together for many different tools over several years, and the documentation isn't great. So a lot of this organization's fundamental systems rely on the knowhow of a few people.

The application was often installed or updated during pre-coordinated upgrade window, in collaboration with networks sysadmins, and this was usually done on the weekend. So this work could range from a few hours to a half dozen of hours. But there have been a lot of instances where the team has had to work throughout the weekend in order to debug or fix a problem because they were doing everything manually, and they were installing so many of these components manually. And if they miss even one config file or anything, they would have to manually debug it, and make sure that the application comes up. So that is what we were trying to automate and increase the reliability that it will not fail, on manual errors at least.

So there was no testing systems, that's for sure. And have we all worked at the same place? This sounds so familiar. We've all been here. He must be a former coworker because this sounds familiar.

Well, this is a project that they're just wrapping up now, I think.

Oh.

Where they're, it's in progress and they're either just finishing it or they just recently finished. I'm not entirely sure on the details, but it's one that's been a long time coming. This process was not a recipe for a good time for those teams working for that client. Right? There had been attempts to automate this process in the past, but they didn't get very far. They had a lot of components that needed to start and stop in a particular sequence. This included a lot of checks along the way to ensure that these components started correctly in the right configuration. Satyajit's team had its work cut out for them.

Sounds like a case for automation. This is textbook right here. This is textbook.

For this project. We estimated around six months of work, which is quite a long time, but it turned out to be quite an accurate estimates because there were many roadblocks along the way.

Now, one of the biggest things that they had to do initially that took a lot of time was to recreate the client's developer environment to as close as they could, because it was such a complex system. And if they didn't have it close enough and they'd then set up the automation, just when they tried to port that automation over back to the client system, it just wouldn't work, right, because the configuration would be different. The machinery would be different. You're essentially building something that's just not going to be applicable to that situation.

And this is... So, let me see if I understand this, Johan. So, and Angela, maybe you can help me out too. So this is sort of a prime use case for automation. This one is difficult though, because the processes are so complex and a lot of the knowledge is in people's heads. Is that kind of what's going on here? Help me understand the root of the difficulty here.

Well, it sounds like they never had a proper testing environment that mirrored their production environment.

Okay.

So it could be that it's running on disparate hardware. It could be different operating system, different packages. Everything is different. And when you're trying to automate something, you want it to be as close to the production system as possible. Any small deviation could be problematic. Right? We can see that if these aren't the same systems and you're trying to build a workflow that works for this, you could possibly have problems when you're trying to test it and say, "Oh, it works in development," and then you try to get it to prod and all heck breaks loose because there was something really very small, but it did deviate. So it's hard when you don't have those mirrored environments to do this type of testing and it can bite you. I mean, as we can see, if it's a really complex system, it's hard to line up all the stars to get it right.

And the complexity came from it's a system that they built up over a really, really long time.

Long time.

Right?

Yeah.

And when we say system, but it's not like it's one computer. It's cobbled together from different pieces of hardware, different pieces of software that they've added to over the years. And whenever they had to run an update, all of these things had to be updated in concert with each other. And it's like conducting a symphony, right, where you've got all these different pieces moving together, and if one piece is off, then-

Yeah.

... it doesn't sound right and it's not going to come out. So they were having to update these and because it's a financial institution, they didn't have a lot of downtime where they'd be able to figure something out if something-

Basically have a weekend, right?

Yeah. They'd have the weekend to update something, so they'd start it and hopefully everything went right, they'd be done in a couple hours, which again is a pretty long time already. But if something went wrong, they'd have to get those systems back online by Monday morning. That's it.

No pressure.

Yeah, no pressure at all. Right? And actually this customer, they got in a little bit of trouble because even though they tried to automate things in the past, they had found it too difficult and they kept making mistakes. They ended up being told to automate the system because of the frequency of the mistakes that they were making.

Oh, wow.

And so that's when they brought the team that Satyajit is on to help them do that.

So this is twofold to me. They were lacking automation that they so desperately needed. But this is also a great use case for app modernization. Think about what you just said. They're talking about disparate hardware and old systems and cobbled together, and those words you hear sounds to me like, "Wow, they would really benefit from something like containerization on top of automation." To get dinged like that though is pretty scary stuff.

Yeah, how'd the project go? How's it going now?

Well, when I spoke to Satyajit, he told me that it had been going really well. It still took a while, still took months of work, but they're getting to that finish line. Now thankfully, the client is getting a lot of benefits from this project.

This automation that we did was more to increase the reliability and security more than speed. Speed is a byproduct, but it was more to increase the reliability and security. And my architect, Jeff, he always gives an example of a robot arm. So when you install a robot arm, it's not only to save time, but it is also to have the exact build at the exact spot.

So that robot arm is providing computer-exact efficiency and consistency. Right? You're getting that exact same result every single time because this is what they needed. They needed to have a system where that they could run it without worrying about one little thing derailing a multi-hour process, and then having to go back and figure out what went wrong and rebuilding everything.

They needed reliability. And security, like he said.

Yeah.

But speed is a byproduct of that.

Yeah, yeah.

For sure. Once you've developed a workflow that can move through this system accurately and efficiently to do everything that you needed, that's where the speed comes in. You're not doing these things manually anymore. So this is a good story. I'm glad there was a happy ending here.

Yeah.

For sure.

Yeah.

Yeah, absolutely.

Because yikes.

So I'm thinking now about both Patrick and Satyajit, and I'm thinking about the difference in scale between the two types of automation that they were working on. And I'm curious what they have in common.

Well, automation allows you to remove the manual processes, and I think that's what they were both dealing with before they brought automation into the fold. And we know manual processes lead to errors and mistakes and issues. So what they do share is that they said, "We're going to automate this issue. Yes, it could be the small issue. Yes, it could be this big issue, but what we're going to do is we're going to take away that error proneness that we were plagued with all along. And what we're going to do is we're going to create this workflow where we can get things done consistently, automatically, time and time again," and they wouldn't have to think about it again. So that's what they share in common. Automation provides you with that safety where it's like, "Oh, okay, I did this thing. I'm not making the mistakes that I've made in the past. I've taken my hands off of the wheels and I'm letting the automation take care of itself."

Johann, how are you thinking about automation now?

Well, when I first started researching this episode, it was very much in that frame of what Jose had said, which is you try to automate something sometimes and you get down that rabbit hole and you're not actually saving all that much time, because it's something that would take you a few seconds and you spend hours automating it. And if you were actually to account for the time writing the automation and actual time doing the thing by hand, maybe that balance doesn't look so good in the end. But speaking to Jose, again, speaking to Patrick, speaking to Satyajit, it really opened my eyes to some of the other things that can be gained from automation, just like Angela was just saying, which is you get that consist... Consistability.

I like that word.

Reliability and consistency all at the same time.

Yes, there you go!

Just put it all together.

Boom. You get that consistency, you get that reliability. And from those two things you get things like speed, increased security, time to focus on something else. And really it's, whether it's just automating little things that make your life a little bit easier to these huge enterprise systems, that's just a matter of scale at that point.

Indeed.

There's one more thing I think we should address here-

Okay.

Which is we heard from Jose about keeping things simple.

Yeah, okay.

Right? Having things do one thing, and one thing that came up in the conversation with Satyajit is obviously this six month work is for a huge system. And the automation that they came up with, it's just, it's complex, it's doing a lot of different things. But apparently the way in which they set it up is that even though the whole of the automation is very complex, it's split up into little different parts that each do one thing.

Oh.

And so what that then allows them to do is that if they discover a bug in part of the process, it's not something that's going to cascade down and affect all the rest of the other automations and the other modules, the other scripts that they're setting up.

Okay.

They can go in and fix things here and there and it allows them to kind of debug a little bit more in a more compartmental fashion, right? So it makes it a lot easier for them. And then later on for the customer, if they need to change something down the line, they can then go fix those individual parts, more than having to look through the entire thing and fix every single bit of it, when they only need one little change.

Making it modular.

Exactly.

Keeping it simple.

Well, we want to know what you think about this episode. I mean, automation is all the rage. That's all we're talking about. That's all you hear. Automate all the things. We want to hear what you have to say about it. So tweet us at Red Hat, using the hashtag compilerpodcast, and we would love to know about your automation journey and what do you think about it? Let us know. And that does it for this episode of Compiler.

Today's episode was produced by Johan Philippine and Caroline Creaghead.

A big thank you to our guests, Jose Vicente Nunez, Patrick Arminio, and Satyajit Lele.

Victoria Lawton is the irreplaceable component we will never automate.

Never. Our audio engineer is Robyn Edgar. Special thanks to Shawn Cole. Our theme song was composed by Mary Ancheta.

Our audio team includes Leigh Day, Stephanie Wonderlick, Mike Esser, Nick Burns, Aaron Williamson, Karen King, Jared Oats, Rachel Ertel, Devin Pope, Matias Foundez, Mike Compton, Ocean Matthews, Paige Stroud, and Alex Traboulsi.

If you'd like today's episode, please follow the show, rate the show, leave a review, share it with someone you know. It really does help us out.

All right, we will see you next time.

See y'all. Bye.

All right, bye.

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.

Adventures in Automation

Adventures In Automation | Compiler

About the episode

Subscribe

Transcript

More about automation and management

About the show

Compiler

Products & portfolios

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links

Adventures in Automation

Adventures In Automation | Compiler

About the episode

Subscribe

Transcript

More about automation and management

Red Hat introduces a new Ansible Automation Platform self-managed offering on Microsoft Azure.

Connecting your systems using the Insights proxy

Ready To Launch | Compiler

How Do We Make Updates Less Annoying? | Compiler

You Can’t Automate The Difficult Decisions | Code Comments

About the show

Compiler

Products & portfolios

Tools

Try, buy, & sell

Communicate

About Red Hat

Select a language

Red Hat legal and privacy links

Red Hat legal and privacy links