Data Analysis with Omar

Understanding the Modern Data Engineering Stack The Modern Data Engineering Stack In the ever-evolving landscape of data management and analysis, data engineering has undergone a significant transformation in recent years. The advent of big data and the need for scalable, real-time data processing have given rise to a modern data engineering stack that leverages a combination of technologies and architectural patterns to meet the demands of today's data-driven world. At the heart of this transformation, three fundamental data storage and processing approaches have emerged as key players: Data Lake, Data Warehouse, and the hybrid concept known as Data Lakehouse. Each of these plays a crucial role in handling the vast and diverse data sources that organizations encounter in their quest for actionable insights. Now, let's delve into these three data storage and processing paradigms and explore the differences, strengths, and best-use cases for Data Lak...

The Reality of Data-Related Jobs The Reality of Data-Related Jobs The hype surrounding data science and data-related roles has created a misconception that everyone is eager to learn about data science, but the reality is quite different. Many individuals continue to work with data on a daily basis without possessing even the most basic data literacy skills. The idea that everyone is becoming data-oriented is often exaggerated. For those aspiring to become data scientists, analysts, or engineers, here's what you should expect at your first job in an average company (not a FAANG/MAANG): Data Literacy It might be expected that everyone working with data at a company would have at least a basic level of data literacy. However, in reality, this is often not the case. Many mid-career professionals with decent pay do not prioritize upskilling in data-related areas. Unless they are compelled to do so, employees tend to stick with their Excel files and rely heavily ...

I bet you’ve watched one of those ‘very useful’ YouTube videos, where you’d be expecting to see what a data analyst do on their daily life (AT WORK), but it ends up being ‘the cool life of a data analyst’. The video starts with a bunch of useless visual effects of the “data analyst” waking up in their half-a-million-dollars bedroom. The best part is that they always wake up with 500 cameras capturing their first seconds of the day, from 500 different angles. They wake up very excited about their day. They are so excited, that the first thing they do is a full 30 mins workout and then they have one of the most luxurious breakfasts you’ll ever see in your life. Finally, and most importantly, they take their shower and right away they start their first ‘business meeting’ where they discuss with 200 executives what they will be doing for the rest of the day. Sounds familiar? Yes… The sad story is that none of this is real. This is more fictional than the ...

Many people approaching the field of data science and data analysis, ask about programming languages. The two most frequent advice are to study either R or Python. However, saying Python without much context can be unobliging and lead the aspiring data analyst or scientist to lose their time on learning unneeded tools. To each field its tools and priorities: the following will explain what is required to start your journey as a data analyst with Python.  Why learn Python in 2021? Python is a general/multi-purpose programming language. You can do everything related to coding on Python: software development (engineering), games development, data science and data analysis, dashboarding (and many more…). In addition, python is relatively simple to learn due to its English-like syntax. In my humble opinion, Python should be learned for the sake of understanding what coding is about, even if you will not use it daily. How to learn Python in 2021? (For beginner...

Data-related jobs have been gaining popularity in the past few years. Many students are aspiring to become data analysts or data scientists; either by taking courses at their universities that relate to those topics (Data mining, machine learning…) or by taking (free or paid) online (coding or theoretical) courses to acquire certifications as proof of their expertise in the domain. Similarly, many professionals are also following the trend.  A career shift into the field of data has become the new normal. However, many do not know how to approach this field after being certified and how to increase their chances of landing their dream job. Therefore, in the following, I will explain what is the easiest way to get a first hands-on experience in the field of data. This will help you highlight your newly acquired skills, from the online coding platforms or theoretical MOOCS that you have taken. What is a portfolio? The term portfolio is highly popular in the arts and creative industry....

Many fresh graduates dream of starting their careers at a Big 4. Their main reasons to join such companies are the title, the money, and the gained experience from working with executives. In addition, the Big 4 companies provide exposure to diverse fields and sectors. Therefore, the work is highly dynamic and less repetitive when compared to the average company.  In the following, I share my personal experience at a Big 4 and what to expect in general when working there. You should read if You are applying or looking to apply for a job at a big 4. Especially if the position is related to data analytics . Who are the Big 4? The Big four refers to the biggest four accounting firms and they are PwC (PricewaterhouseCoopers), KPMG ( Klynveld Peat Marwick Goerdeler ), and EY (Ernest & Young). They are also famous for their consulting departments which are the dream job for many fresh graduates from diverse fields (Business/Econ, STEM, Law...)...

Data-related jobs (data science, data analyst, data engineering…) are the hottest jobs of this decade. Many companies are upgrading their systems so they can benefit from their data. In addition, many people are aspiring to become part of this data revolution; mainly because the jobs are relatively fun and pay high wages. In the following, I share my two cents on this topic mainly due to the questions I see online on whether to pursue a Master’s degree in data and if it is worth it to secure the job. You should read it if You are looking to get into the data field or if you are already in the data field but looking for an extra challenge. Master of Artificial Intelligence When I first applied to my master of Artificial Intelligence, my ultimate goal was to combine what I have learned as a Civil Engineer (in transportation specifically) with what the MoAI is going to offer me. However, I noticed that the job market combining AI and CE is a niche market, where opportunities are either ...

Motivation... Many data analysts create functions or scripts to automate the cleaning of files that they believe come in the same format. However, human mistakes are pretty common when performing data entry. On my daily work I deal with "supposedly" identical datasets, however, the script would run for 2 to 3 files before throwing an error showing that a file is not in the same shape as the others.  Who should read it Any data practitioner, in general. Or anyone looking to enter the field sooner or later. Data Inconsistency In general The biggest frustration for data analysts and scientists is data inconsistency. It adds an extra layer of suffering to our daily work. Why? Because with inconsistent data, not only we have to transform the data into the shape we actually need, but we also have to clean all the mistakes done by others. In short, a clean data that would usually take 5-10mins to put into good shape for modelling or visualization (pivot, stack unstack...), now requi...

Many persons want to become data analysts and scientists. However, not too many people working in that field are telling exactly what to expect. Most videos on YouTube are very vague and made using datasets from Kaggle, which are in many cases clean (general structure), but require filling missing values. At work, the data is ugly . In the following we will discuss why... Who should Read it... Anyone looking to start their career as a Data Analyst and co. How to Read it Fully. Or watch the video on Youtube. Dataset The dataset used in the following, can be considered as something to be expected from a client. It is a real life template, with all the values changed for privacy purposes. Now the issues with this data is that it is considered wide. So it isn't possible to visualize it on Python unless some transformations are applied to it. The ultimate goal is: To extract the year from "Period_2020", th...