I currently started work as a data analyst. For a data analyst, the work is quite programming heavy actually (e.g. we don't use Excel but Jupyter). But for a software engineer, the amount of programming feels low. It's more like creating quick scripts.
For now, I'm having a blast but I want to keep my actual software engineering skills sharp where I also learn some stuff on a part-time basis. Preferably stuff that I can also put on a resume that will be seen as experience.
What would you suggest I should do? I was thinking about joining an open-source project of a big enough scope or something. Perhaps I should insist on fun software, which would be certain web apps or game development in my case.
I think one of the most toxic things in this industry is that we think we need to continuously be doing coding outside of work instead of living life.
So the answer is I don’t.
I had been programming continuously from the time I was in 6th grade in 1986 until 2020.
As I pivoted into more strategic cloud consulting specializing in cloud architecture + application development in 2020 - “cloud application modernization” - I started coding less and less to the point now that almost any time that I even suggest writing code at work even though my title is “staff software architect”, I get my hand slapped and asked why didn’t I delegate it and focus on “higher impact work”.
I keep my knowledge of the industry up to date by continuously learning during the work day. But when I get off work, I shut my computer down and live my life - spending time with my wife and friends, exercise, travel, concerts, etc.
We travel - a lot. One of the benefits of remote work.
If I do need to pivot back into hands on coding, I have a years worth of expenses in savings in the bank aside from retirement savings to give me time to get back into the groove of things.
My career started really rough, first 1.5 years didn't get a job and I have another 1 year gap. I don't have that luxury. I have experiences in stacks for 6 months tops with most. If it's longer it's an esoteric language. No one cares about my CS degree(s) and the experience I gained there, despite the fact I learned more in some courses than I did in most jobs.
This is the first job I can fully showcase all my skills, so that's amazingly good. In that sense I need to seriously reconsider how I can make a career out of being a data analyst. But I'm not that far in my thinking yet and still see myself as a software engineer for my next job and while this job gives me great skills to learn in marketing and soft skills (and some stats and some dev), it doesn't give enough engineering skills.
As a SWE I'm in an odd place. I have never been in a place where SOLID is a thing. I don't know design patterns that well (I can recognize them and in many cases they make sense). But I have engineered quite a bit and I have seen enough 1 million LoC or bigger codebases.
So I'm still in a bit of a weird in between state and could use some mentoring with regards to good software design and just good ol' engineering, which is why I was thinking to help out with some open-source software or something like that.
I agree that the "always be programming" culture is toxic. It is what it is. I'm the type that just rolls with the punches. Much of my life has been on hard mode for a variety of reasons, so a toxic culture like "always be programming" is honestly not the worst thing.
As much as I rail against it, I realize the privileged position I’m in now as a 50 year old, strong experience and a stint at BigTech on my resume.
If I were just starting out in my career, I would be spending whatever time it took to get into BigTech and adjacent companies - yes “grind leetCode and work for a FAANG” (tm r/cscareerquestions) and I would also be learning good software engineering practices and some g framework to be prepared for techno-trivia interviews on my chosen stack.
I guess my point is, there is a huge difference unfortunately between being prepared for a job and being prepared to interview. Contributing to open source doesn’t help the latter.
> I think one of the most toxic things in this industry is that we think we need to continuously be doing coding outside of work instead of living life.
Why is that toxic and what other industries are you comparing that to?
Do fun software, games especially - you'll learn heaps.
Don't do open source for the sake of it, unless its a product you actually use.
Perhaps you could fork my simple raycaster and add your own enhancements to it, its a simple project [0]
[0] - https://github.com/con-dog/2.5D-raycasting-engine/blob/maste...
The only solution I found working is to leave data field and get back to software one (within same org or not).
No other reliable and meaningful way.
Nothing beats actual job in improving a complex skill. There is no shortcut to imitate that.
This. I've been in a similar situation working as a data scientist for 2.5 years and went back to software engineering a couple of years ago.
I'm not trying to say that data science and analytics are necessarily bad environments; I just came to realize that I had different expectations for my work than my organization.
While the skills of a software engineer — e.g. quickly comprehending technical references, operating APIs, ability to type more than three lines of code straight — are highly valuable for a data analyst to be productive, I had to realize that my organization did not appreciate the craftsmanship of producing code as much as I still do.
Over time, I had to witness my analyses end up on slides or in Excel workbooks, only to be looked at once. They'd done their job and weren't needed anymore. I was effectively "prompt completing" analytic requests from management to understand the organization/business/whatever better — always with the same result for "my work".
Providing an organization with the intelligence to understand their business is for sure not a bad motivation and can be fulfilling. It's just not a good fit if — instead of the analytics — you consider the software you create for the analytics as your work.
I agree with this. At some point you need to work on a hard project, with minimal support, for an extended period of time. The skills you'll learn will be indispensable. Open source can help, but it's hard to replicate being immersed in a difficult role for 40 hours per week.
The more important thing is that you have a genuine interest in the field, which should take you where you need to go. But without at least a small amount of hard software experience you'll be stuck at an intermediate level.
Aren't certain open-source projects like a job?
Only if you work on it more or less full time and have colleagues or someone holds your accountable for the outcomes. Otherwise it's a hobby.
AI has made some once absurd ideas possible. For example, you can hardcode personal details and have a resume builder export it to PDF or a web portfolio. It'll probably be less work than formatting a resume in Word. Leave AI to do repetitive work like mapping data types, while you figure out the code to make it look pretty.
The Missing Semester of Your CS Education: https://missing.csail.mit.edu/ I would say finishing this helped me a lot to learn better and build up on tech stuff I use at my day job. This is a nice short course giving a very useful intro to software engineering tools required upon which you can build up further on the go, otherwise it's very easy to miss those topics and stay an 1x developer.
Could you elaborate on this a bit?
> But for a software engineer, the amount of programming feels low. It's more like creating quick scripts.
I feel like it’s the other way around. When I worked as a data analyst (before transitioning to a data scientist and eventually a full-fledged SWE), my experience was quite different. As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging.
On the other hand, SWE is a different world. I guess it depends on the type of SWE you are and the nature of the organization you work for. At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA.
That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1]. Since I have to publish something each week, it forces me to think about different problems and seek out new technologies to try out so I can write about them. I also write about things I’ve figured out at work, which has been immensely helpful. Over the years, this act of actively seeking knowledge has added up and become substantial.
More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms.
Your approach might be different. Find something tangential to programming and work on it outside of work hours. Even a few hours per week adds up. For me, it’s side projects and writing. For you, it might be something else.
[1]: https://rednafi.com
I think I explained myself poorly.
> At a large-scale organization, as an SWE, I did far more programming work than I ever did as a DA.
Yea, this is what I meant. The amount of programming I do now as a DA feels low (since I'm a software engineer historically, that's what I meant with "but for a software engineer" - a bit of a confusing sentence). Well specifically, the amount of engineering feels low.
> That said, there are different ways to cope and build skills—it takes time. I do side projects and, more importantly, write about them regularly [1].
Good point, to what extent do employers see this as experience though? It's my experience they don't care at all. Maybe I'm looking at this wrong. The reason I'd do it is (1) I want to keep on building and (2) I want to keep on building experience as a SWE that employers will be impressed by. I'm not sure if I want to switch back, but for now I want to keep that option open.
> More than once, people have reached out with job offers just because they stumbled across my writing on Hacker News or other social media platforms.
I guess you answered your question here. I think it's good enough to test it out :)
> As a DA, I mostly wrote aggregation queries, built dashboards, and did some Python, Pandas, and Notebook munging.
Yea, I see a lot of SQL queries here, notebook stuff (so also Python/Pandas) and dashboarding. It's probably a 50/50 split. I have a suspicion that my manager wants me more on the programming side of things though. So far me as well. I'm allowed to pick any tool I find right for the job. I already wrote a bash script today.
> Good point, to what extent do employers see this as experience though?
They don’t see it as experience.
> I'm not sure if I want to switch back, but for now I want to keep that option open.
For the most part once you start going deep down one rabbit hole, it’s almost impossible to change jobs and go back to software development. I see it happen all of the time with people who leave software development and go into “DevOps” [sic].
Companies aren’t going to take your side projects seriously. With the current job market, why wouldn’t they hire someone with recent relevant experience? If you can get an interview with a company that relies heavily on coding tests you might stand a chance.
The best way to pivot back is to get a job where you do have recent experience - in your case data - and then after awhile do a formal or informal transfer to more of a software engineering focus
> If you can get an interview with a company that relies heavily on coding tests you might stand a chance.
Fair point. I think if I grind leetcode full-time for 3 months, I'll be there. I have around 250 questions under my belt (50 hard), and I feel definitely familiar nowadays. I'm mostly in the refining part, sometimes still in the "I don't see the trick" part.
So the question becomes how do you get to the interview and stand out among the hundreds of people applying for the same position?
I don’t have a good answer for that as a software developer. I looked for bog standard remote development jobs as a Plan B with 20+ years of development and deep AWS experience last year and the year before and it is a shit show out there now.
Luckily, I have deep experience in my niche - strategic cloud consulting + app dev and a stint working at a BigTech company in the consulting department that opened a lot of doors.
I think the play may be to stay in analytics and get into a well known company.
Also, make sure you are taking on assignments with bigger “scope” and higher “impact”. Don’t be a ticket taker.
> I think the play may be to stay in analytics and get into a well known company.
> Also, make sure you are taking on assignments with bigger “scope” and higher “impact”. Don’t be a ticket taker.
Solid advice, thank you :)
I was a ticket taker. The irony is that I'm not even a week in on my first job and this manager is definitely not making me be a ticket taker. I already completed some scraping projects. Not sure where that falls as being a "data analyst" but yea the data analysts are quite technical anyway (i.e. no one uses Excel, it's all SQL and Python).
> Good point. To what extent do employers even see this as experience, though? In my experience, they don’t care at all. Maybe I’m looking at this wrong.
I don’t know to what extent this is considered experience either. That said, the hiring process in tech is completely broken. There are companies that ask LeetCode hard questions for positions where you’ll just write HTML templates and React.
I work on side projects and write because I enjoy the process—not because I’m fishing for potential employers. In the past half-decade, I’ve written close to 200 articles. Researching for them was immensely helpful in building knowledge. At the start, I didn’t care if anyone read them. But eventually, the site got traction, and now it forces me to try new things and keep my skillset sharp. This works as a positive feedback loop.
Ah I see, thanks for the perspective. Follow your bliss, get traction ;-)
If you're looking for suggestions outside of work, one important factor will be motivation. You're not paid for it so you'll need something that will hold your attention regardless, or you won't commit to it. Therefore, the answer depends on what you personally find interesting.
Yea I thought about that, probably game dev from the ground (with my own graphics engine) up or a web app that I find useful (currently no thoughts on that).
As someone in a similar situation, my number one piece of advice is to meticulously keep up your software engineering best practices even when doing your day to day data analysis job. All that bread-and-butter stuff like tests and easy deployment and well structured code and reproducible builds and writing reusable tools and packages tends to fall by the wayside when your 'just' hacking one off Jupyter notebooks.
As a software engineer you bring a perspective to the job that many of your colleagues may lack. Lean on that and use that background to help build better tools and ways of working. This not only makes your job (and the job of your colleagues) easier in the long run it will also let you keep your software engineering skills sharp.
> even when doing your day to day data analysis job
This is a bad advice.
Usually coming from an over-engineering mentality (that many engineers suffer from, including myself).
Part of the engineering culture is to find the _right_ solution for the job. Not the most _engineered_ one.
There is no point to shoot at sparrows from a canon.
How is this bad advice?
> keep up your software engineering best practices
It's not a best practice to over-engineer solutions, and indeed, it is a best practice to find the right solution.
When your job is to cobble up disposable scripts - there is no place for “practices”, unit test for example. Thus there is nothing for a software engineer to pay special attention to. Therefore there is no value in suggesting that
While some may take that as “turn up your engineering knob” that leads to over engineering
While I am fairly good at overengineering and optimizing, I have always hated doing it. For the longest time I wasn't able to do it but then I learned about low level hacking, and throwing things into assembly. Then I had a friend enter the HFT field and he taught me a thing or two about optimization.
godbolt.org is fun to check out :)
But yea, I really like making quick scrappy prototypes that deliver business results quickly. I've always had that mindset rather than crafting good quality code. In software engineering teams I threw that mindset to the wayside because they'd get mad. Sometimes rightly so as tests have saved my life, but quite often I think SWEs focus too much on code and being technical rather than delivering impact. I think in part that's also because it seems that in most places I've worked at product owners are "supposed to" think about that. The dev is only supposed to think about the technical implications, maybe the UX implications (depending on the company) but the moment a dev thinks about business implications, I've gotten quite a few times to "stay in my lane".
In most cases the care for code quality isn't needed, sometimes it is. I'm taking a page from Jonathan Blow here (the game dev that created The Witness and Braid). He'd develop everything unoptimized and only later when he'd run into performance issues would he optimize it.
If you write a lot of "quick scripts", you might be writing a lot of the same boilerplate code over and over again. If that's the case, creating a framework/library that does X/Y/Z (related to your job) might help.
Yea that's fair, just started, so we'll see. Thanks for the tip though!
If you’re interested in security-related self-studies, I can warmly recommend Cryptopals: https://cryptopals.com/
That brings up a good point! I could also go the hackthebox.eu route to train my security engineering skills, including cryptopals.
PlayerTwo was still a fun insane box :)