GCP DATA ENGINEERING BY VIJAY SIR DURGASOFT YOUTUBE LIVE STREAM

Added: 2026-05-08

[00:00:16]Yes.

[00:00:20]Yeah, good evening all. Right.

[00:00:23]So, myself I'm Vijay. Like I hold 20 plus years of experience.

[00:00:28]My experience including different technologies like big data, Hadoop, Spark, PySpark and this GCP data engineer, right? Yes.

[00:00:41]So, before I start, can anyone please confirm whether my voice is clear, whether the screen is visible or not?

[00:00:46]Yes.

[00:00:49]Is my voice clear to all? Is my screen visible to all? Right, fine.

[00:00:52]>> Yes.

[00:00:54]Yes.

[00:00:56]As part of today's demo, I'll be covering the following things, right?

[00:01:00]>> [snorts] >> Who is a data engineer?

[00:01:11]Means how a data engineer is different from a traditional ETL developer?

[00:01:16]How is different from a traditional ETL developer? Data engineer roles and responsibilities.

[00:01:22]What are the various data engineer roles and responsibilities?

[00:01:27]And what is Google Cloud Platform?

[00:01:30]Various GCP data engineering products or services.

[00:01:36]Job market of GCP.

[00:01:44]Yes, fine.

[00:01:46]>> [snorts] >> So, previously, if you go back like 10 to 15 years, right? If you go back like 10 to 15 years, there is no concept of data engineer. There is no concept of data engineer.

[00:01:57]We used to have ETL developer.

[00:02:00]We used to have ETL developer, right?

[00:02:01]Yes.

[00:02:03]>> [snorts] >> But how a data engineer is different from a So, the tool here we are using GCP, the cloud GCP, right?

[00:02:14]Yes. And if you talk advantage of cloud, if we talk if we're talking about a cloud, advantage of cloud over this on-premises.

[00:02:27]So, here in the case of on-premises, we need to set up our own hardware infrastructure.

[00:02:34]We need to set up our own hardware infrastructure.

[00:02:39]On top of that, OS installation.

[00:02:43]Purchasing of database license.

[00:02:46]Purchasing of database license.

[00:02:51]Designing of database objects. Designing of database objects.

[00:02:57]Yes.

[00:03:00]To maintain all this, we require separated teams.

[00:03:04]To maintain all this, we require separated teams like hardware team is required.

[00:03:11]Operating system team is required.

[00:03:14]Networking team is required.

[00:03:16]Security team is required.

[00:03:20]Database administrators. But here, everything in the cloud premises. But here, everything in the cloud premises, we no need to maintain all this. No need of No need to maintain any infrastructure. No need to maintain any licenses.

[00:03:35]Everything in the cloud. Here, everything in the cloud.

[00:03:41]>> [snorts] >> Yes.

[00:03:46]Advantage of cloud over this on-premises, right? Yes.

[00:03:51]If you talk, we have got many cloud service providers. We have got many cloud service providers.

[00:04:01]Okay. Now, see this various cloud service providers. AWS, Azure, GCP, Oracle Cloud, Salesforce, VMware, Liquid Web, Alibaba, Rackspace, Digital Ocean.

[00:04:23]But most of the companies are preferring GCP, Azure, AWS, right? Yes.

[00:04:29]Why GCP? Why we require GCP?

[00:04:33]As compared with Azure or AWS.

[00:04:42]See here.

[00:04:44]GCP growth is increasing rapidly.

[00:04:47]GCP growth is increasing rapidly.

[00:04:50]Here, we store data.

[00:04:53]Here, we retrieve or read data.

[00:04:57]We perform data analytics.

[00:05:00]We see high availability of data. We see high availability of data. Data is stored in different places.

[00:05:06]If data is lost at one place, we can see the data in other, right?

[00:05:11]>> [snorts] >> Data stored in different places.

[00:05:15]If one server is down, we get data from other server.

[00:05:19]Your data will be safe and secure.

[00:05:26]Okay.

[00:05:27]Then, easy to scale.

[00:05:30]Easy to scale. We can increase or decrease as per our requirements.

[00:05:35]Easy to scale. We can increase or decrease as per our requirement.

[00:05:39]GCP offers wide scope of applications.

[00:05:44]GCP offers wide scope of applications to run web applications, data analytics, machine learning, right? Yes.

[00:06:03]Why to choose GCP?

[00:06:05]Why to choose GCP as compared with Azure or AWS? Why I need to choose?

[00:06:10]Means one, it's the fastest, biggest network in the world. It is the fastest, biggest network in the world.

[00:06:22]Google has got data centers. Google has got data centers throughout the world.

[00:06:27]They use high-end virtual machines. They use high-end virtual machines.

[00:06:48]Petabyte bandwidth.

[00:06:50]Petabyte bandwidth. High performance.

[00:06:54]Separation of computer and the storage.

[00:06:58]Every individual virtual machine will have its separate RAM and data storage.

[00:07:04]Each cluster will have a number of nodes.

[00:07:07]The number of nodes we can store data.

[00:07:11]If data stored in single node, if it get crashed, then the data is lost.

[00:07:18]So, in GCP, if if one copy is stored in one machine, other copies are stored across multiple machines.

[00:07:26]Here, from different data centers, we can retrieve data.

[00:07:30]From different data centers, we can retrieve data.

[00:07:38]GCP has flexible pricing.

[00:07:41]GCP has flexible pricing as compared with AWS and Azure.

[00:07:52]Customization. See, even the flexible pricing, discounted prices also will be given for running l- bigger workloads.

[00:08:05]For For example, for 5 years, 10 years.

[00:08:08]Customization. Customization.

[00:08:11]Wide range of customizations. For a particular instance, we can customize depending upon the requirement. Like for instance, having 1 petabyte of storage.

[00:08:24]Scalability. Easy to scale.

[00:08:27]Capacity increasing or decreasing.

[00:08:50]Scalability capacity. Easy to scale in capacity increasing or decreasing. Live migration.

[00:08:56]Here, we can have in GCP live migration.

[00:09:01]Supports live migration of virtual machines.

[00:09:04]Live migration of virtual machines.

[00:09:07]Whatever existing system we have, whatever existing system we have, if you store data in if you have got data, you can lively migrate to GCP without interact without impacting the existing data.

[00:09:24]Big data ML.

[00:09:31]We get inbuilt and predefined ML and AI APIs in GCP, right? We have got many inbuilt and predefined ML and AI APIs in GCP.

[00:09:57]See >> Free trials, some services free for 3 months.

[00:10:02]Some services free for 1 year.

[00:10:05]Some services free for lifetime.

[00:10:09]Some services free for 3 months, some services free for 1 year.

[00:10:13]Some services free for lifetime.

[00:10:16]Customer systems migration. Customer systems migration.

[00:10:31]Existing data migration to GCP. Existing system existing data migration to GCP.

[00:10:40]Like we've got HSBC, Twitter, New New York Times, Spotify, Vodafone, all these things, right?

[00:10:50]They're migrating to GCP.

[00:10:56]So here I was talking about GCP, right?

[00:10:58]Yes.

[00:11:00]So this is that why we choose GCP. These are the major things we see.

[00:11:04]The mainly it's the fastest, biggest network in the world. It has got petabyte bandwidth.

[00:11:10]It has got flexible pricing. It has got scalability.

[00:11:22]Live migration we can say live migration.

[00:11:28]Free trials.

[00:11:31]Customer systems migration, right? Yes.

[00:11:35]Okay, data engineering involves data engineering, everything deals with the data.

[00:11:42]Here in data engineering everything deals with the data.

[00:11:47]Total world is flooded with user data. Using data only we perform everything.

[00:11:54]Data.

[00:11:56]Using data we perform data analysis.

[00:12:01]Data has got experience. Data has got future reference.

[00:12:13]Using data or data science we perform data analysis.

[00:12:17]Okay. So here we perform using data engineering like data handling data, ingesting data, or extracting data, or storing data, filtering data, transforming data, processing data, merging data, sorting data, grouping data, analyzing data.

[00:12:40]So these are the various things in data engineering, right? Mainly mean loading.

[00:12:54]ETL Data engineering involves How how we extract the data? How to extract the data? Check it.

[00:13:05]Means from where What are the sources from where we can extract the data from?

[00:13:10]So what are the sources of the data?

[00:13:12]From which sources we need to extract the data?

[00:13:15]How to process the data?

[00:13:23]And what is the destination of the data?

[00:13:25]What is the destination of the data?

[00:13:32]How to extract data? What are the sources of the data? How to process data?

[00:13:39]And what is the destination of the data?

[00:13:44]A data engineer designs and maintains an organization's entire data infrastructure.

[00:14:06]Mhm.

[00:14:28]Data engineers.

[00:15:03]Data engineers with the help of cloud services they work on the data and make it structured and useful.

[00:15:25]A data engineer is responsible for smooth data flow.

[00:15:30]Smooth smooth data flow.

[00:15:35]From source to destination.

[00:15:41]Data engineer is responsible for smooth data flow from source to destination, right? If you talk Data How data engineer is different from a data scientist?

[00:15:53]Data scientist is going to strategize for the future.

[00:16:14]Analyze the data.

[00:16:19]>> [snorts] >> Identify the key patterns.

[00:16:29]Analyze the data.

[00:16:49]Recommendations for the future.

[00:16:54]Strategize the future.

[00:16:58]Analyze the data, identify the key patterns, recommendations for the future.

[00:17:29]Strategize for the future, analyze the data, identifying the key patterns, recommendations for the future.

[00:17:35]A data scientist.

[00:17:38]So he performs data scientist Data engineer.

[00:17:50]Data engineer.

[00:17:53]But data engineer.

[00:18:05]Mainly for data storage plus data management.

[00:18:32]Data scientist.

[00:18:34]>> [snorts] >> Data scientist does many things, right?

[00:18:36]Data scientist If you If you want data scientist.

[00:18:57]Data analysis. He performs data analysis.

[00:19:04]>> [cough] >> Excuse me.

[00:19:10]Data predictions.

[00:19:23]Data classification.

[00:19:27]For data analysis, data predictions, data classification.

[00:19:32]Data clustering.

[00:19:42]Data recommendation.

[00:19:45]So data analysis, data predictions, data classification, data clustering, data >> [snorts] >> Okay.

[00:20:19]And data Okay.

[00:20:22]Another side data analyst.

[00:20:30]Summarization will revise the data of the past.

[00:20:33]Data engineer for data management.

[00:20:36]Data storage and data processing.

[00:20:39]Sorry.

[00:20:41]Again, data engineering for data management.

[00:20:45]For data storage and data processing.

[00:20:50]So, data engineer should be able to work with the structured data.

[00:20:54]Should be able to work with semi-structured data. Should be able to work with unstructured data. Like he should be able to work with streaming data with data which keeps on incrementing, which keeps on generating.

[00:21:05]Should be able to work with the no SQL data, not only SQL, right?

[00:21:10]Schema-less behavior, random access.

[00:21:25]>> [snorts] >> Okay.

[00:21:32]You should be able to work and you will be able to work with all the file formats. You should be able to work with all the file formats.

[00:21:39]Like CSV format, JSON format, parquet format, XML format. You should be able to work with all the format.

[00:21:48]So, here if you see So, here data scientist, data engineering and data analyst, right?

[00:22:08]Data engineer does all the groundwork Data engineer does all the groundwork for data scientist and analyst.

[00:22:44]More than 60% of data analysis Data analysis work is done what?

[00:22:59]Yes.

[00:23:09]More than 60% of data analysis work Data engineers.

[00:23:28]So, now coming to this GCP, right? Okay.

[00:23:32]So, Google Cloud Platform.

[00:23:34]It's a computing service.

[00:23:38]It allows the customers to have PAS, Platform as a Service, Infrastructure as a Service and serverless compute.

[00:23:47]>> [snorts] >> And the a typical data engineer A GCP data engineer A designs A GCP data engineer designs, builds and maintains data pipelines using Google Cloud.

[00:24:02]>> [snorts] >> Is going to design, build and maintains data pipelines using Google Cloud. A GCP data engineer ensures data quality, optimizes data storage and process data processing, collaborating with the data scientists and analysts, monitoring data infrastructure.

[00:24:27]Monitoring data infrastructure. See, GCP data engineer, right?

[00:24:33]Is going to maintain the entire organization's data infrastructure, complex data pipelines, big data technologies.

[00:24:44]And while if you talk about ETL developer focuses more on the extract ETL, extract, transform and load process.

[00:24:56]Data engineer has a broader, more architectural role utilizing the big data platforms and cloud services, whereas the ETL developer, whereas the ETL developer has a more specialized role of using traditional ETL tools and this Okay.

[00:25:17]>> [snorts] [snorts] >> Okay.

[00:25:35]And what are the various GCP data engineering products and services?

[00:25:40]Yes.

[00:25:51]We have got uh Google storage for storing data. BigQuery for fast querying BigQuery, right?

[00:26:05]For fast querying, for storage and analysis of large data sets.

[00:26:09]We have got DataProc. We'll be discussing about DataProc for big data processing and analytics.

[00:26:16]Data bricks for PySpark processing in GCP.

[00:26:20]Data flow for stream and batch data processing.

[00:26:25]And Cloud Composer, Apache Airflow for automating and monitoring data pipelines.

[00:26:32]Pub/Sub for messaging and data ingestions.

[00:26:39]You will get total idea, right? Once you can go through this course content once.

[00:26:43]What are the things we'll be discussing?

[00:26:45]You'll GCP data engineering module one, GCP data engineering fundamentals.

[00:26:53]Google Cloud Storage, Cloud SQL, setting up a database.

[00:27:00]BigQuery for building a for building data warehouse.

[00:27:05]DataProc, big data processing.

[00:27:09]Data bricks, PySpark processing.

[00:27:12]Data flow, Apache Beam development.

[00:27:16]Google Cloud Composer, orchestration.

[00:27:21]Data fusion.

[00:27:23]Data integration service.

[00:27:25]DBT, data build tool.

[00:27:28]Airflow and Terraform.

[00:27:31]So, majorly these 10 things will be discussed, right? In brief.

[00:27:35]Means if we see the introduction to this Google Cloud Platform, overview of cloud platforms, GCP data engineering fundamentals.

[00:27:47]GCP getting started with GCP.

[00:27:51]Creating GCP account.

[00:27:53]Getting a new Google Google account using non-Gmail ID.

[00:28:00]Overview of credits and overview of this price and billings.

[00:28:10]Storage and databases in GCP.

[00:28:13]Storage and databases in GCP, right?

[00:28:15]Yes.

[00:28:16]Cloud Storage, BigQuery, Cloud SQL, Cloud Spanner for relational databases. Firestore, BigTable for no SQL databases.

[00:28:30]And data modeling best practices.

[00:28:36]Google Cloud Storage, GCS.

[00:28:39]Getting started with this data lake using GCS.

[00:28:44]Google Cloud Storage, right?

[00:28:47]Cloud SQL, set up PostgreSQL database using Cloud SQL.

[00:28:52]Installing of this PostgreSQL, setting up tables in GCP.

[00:28:59]Data ingestions and integration.

[00:29:02]Pub/Sub, Data flow, DataProc.

[00:29:05]Yes. Data processing and transformations.

[00:29:09]Data warehousing and analytics, BigQuery.

[00:29:12]BigQuery for building data warehouse.

[00:29:20]DataProc.

[00:29:22]DataProc.

[00:29:30]Big data processing.

[00:29:37]DataProc, big data processing.

[00:29:45]ETL data pipelines using a data proc data bricks by spark processing in GCP with many working examples ETL pipelines using a data bricks integration of spark and data proc and big query integration of spark and data proc and big query data flow Apache beam development cloud pub sub Google cloud composure for data pipeline orchestration data fusion cloud functions terraform data pipelines using data build tool airflow and big query right these things machine learning in GCP security monitoring and governance real world GCP data engineering scenarios building a stream streaming pipeline building a batch pipeline data migration from on prem to GCP designing hybrid data lake big query data flex and GCS yes project close around two files I'll be discussing as part of this course next see offerings from my side if you see that what are the various offerings from my side you'll be getting each and every live session video means every session is a video recorded means if you miss any session you'll be getting that session video you can go through that and attend the next session for the better understandings soft copy of this class notes assignments are tasks to work a WhatsApp group for technical discussions interview questions around 200 interview questions live interactive hands-on sessions so all the sessions will be interactive at any point you can ask your queries if you want to do the things practically along with me you can do it Python videos apart from this right these are the offerings from my side along from that Durga soft offering right you'll be Python videos access will be provided around 100 Python videos MySQL videos around 35 videos and Linux videos around 100 videos right yes so you can just to go through even I'll be discussing as part of because basic Python and basic SQL are the prerequisites right for this course so that I'll be discussing about the Python syntaxes yeah [snorts] any other queries you got so this is just introduction to this what we'll be discussing and what you'll be learning as part of this course you will get great command again that data engineering is a very if you see the field data engineering field has got great demand data engineers and data scientists has got great demand in today right and that way if you see the salary packages right even if you see 1.5 times more than other technologies you can demand for the salaries so in the same way they'll be expecting the command from you in this or in this technical skills right yes so after the end of this course you a typical 3 plus 4 plus years experience how much knowledge is to gain that much of knowledge you are going to get it's totally like 10 to 12 weekends course around 3 months it takes 12 weekends course 12 weekends course Saturday for 2 hours Sunday for 2 hours totally like 4 hours goes like around 12 weeks it goes some of sometimes I'll be extending the sessions with one more hour any other queries you got anyone right sir you are covering the big query also right yes big query in brief right you can just check this one second means how to write the big queries to face the data this kind of thing right yes you can see in this integration of spark and data proc and big query and also big query for building data warehouse this entire thing you can check it right big query for building this entire thing okay okay means whatever is basic required you are going to provide in big query right yes yes at least we can understand the queries and we can write the big queries to face the data integrations also I'll be discussing okay so everything in brief that's why I'll be spending around 3 months time for this so clear notes will be provided the videos you'll be getting assignments and tasks assignment questions interview questions sir anyhow we can change the time or 5:00 p.m. is the the fixed it initially it will be 5:00 p.m.

[00:35:50]for 2 3 weeks later it will be 6:00 to 8:00 it will be the timing 6:00 to 8:00 okay >> [snorts] >> okay tomorrow also you people can attend the session same time at 5:00 o'clock using the same link to see more discussion yes some yes some query was asked sir here you are covering CCD related thing any or pipeline settings yes okay okay tomorrow also same time just we'll have some introduction but from next week we'll be starting tomorrow just I'll be seeing I'll be discussing how we can interact with the services right what are the different methods of interacting with the services different ways of interacting so we have got multiple methods if you want to interact with through command based through console through SDK what are the different ways what are the differences you'll be observing so from the next week we'll be starting with the actual things right yes okay so if you're done with the queries if there are no other queries I'm signing off for now meet you tomorrow same time at 5:00 using the same link to see more discussion right yes thank you all for your time thank you bye thank thank you thank you thank you sir okay so yesterday we had introductory session in this GCP data engineer who is a data engineer data engineer roles and responsibilities GCP cloud platform various GCP data engineering products or services job market of GCP right job market of GCP okay so just I'll take 5 minutes what I discussed in the last class right advantage of advantage of cloud over on premises right why we are going for cloud as compared with on premises we need to set up our own hardware infrastructure we need to set up our own hardware infrastructure on top of that OS installation we need to set up our own hardware infrastructure and on top of that OS installation right yes purchasing of database license designing a database objects to maintain all this again that we require separate teams like hardware team operating system team networking team security team database administrators but here everything in the cloud premises you no need to maintain all this no need to maintain any infrastructure no need to maintain any license everything in the cloud and everything in the cloud right various cloud service providers AWS Azure GCP Oracle cloud salesforce VMware liquid web Alibaba cloud rackspace digital ocean right most companies are preferring GCP Azure AWS why GCP as compared with Azure or AWS means GCP growth is increasing rapidly from the past 1 year if you see as compared with other clouds GCP growth is increasing rapidly.

[00:39:43]It is the biggest network.

[00:39:45]It is the fastest growing network. Here, we store data.

[00:39:49]We retrieve our data, data. We perform data analytics. We see high availability of the data. If the data is lost, again we can retrieve from different other regions, right?

[00:39:58]Data is safe and data is secure.

[00:40:01]Easy to scale.

[00:40:03]GCP is flexible.

[00:40:05]Why to choose GCP?

[00:40:09]Fastest, the biggest network in the world. GCP is the fastest, biggest network in the world.

[00:40:16]Petabyte bandwidth.

[00:40:20]Separation of compute and storage. Separation of compute and storage.

[00:40:28]Flexible pricing.

[00:40:33]Customization.

[00:40:38]Scalability. Easy to scale. You can increase or decrease as per requirement.

[00:40:43]We can do a lot of customizations, I said.

[00:40:45]As per your requirement. Flexible pricing. Even a lot of discounts are applied if you're going with a huge workloads.

[00:40:54]Live migration. In some of the other clouds, it's not available. In GCP, we can migrate.

[00:41:03]Big data and machine learning. Lot of inbuilt We have got many APIs, inbuilt APIs are available towards big data and machine learning, right?

[00:41:13]Free trials. Some services are free for lifelong. Some free are some are free for 1 year.

[00:41:19]Some are free for 3 months.

[00:41:23]Customer systems migrations.

[00:41:26]So, these [snorts] are the things, right? Which makes GCP popular.

[00:41:31]If you see it, previously I said we have got ETL developer. As compared with the traditional ETL developer with data engineer right? Data engineer has got a lot of responsibilities.

[00:41:43]The data engineering involves handling the total everything. He deals with the data. He keeps on dealing with data, filtering or grouping or merging or sorting or loading of this data, right?

[00:41:55]So, all this he'll be handling all this, handling data, ingesting data, filtering data, transforming, processing, merging, or sorting the data.

[00:42:04]Analyzing. Data engineering involves how to extract the data.

[00:42:09]What are the sources of the data?

[00:42:11]And how to process the data.

[00:42:14]And what is the destination of the data?

[00:42:17]A data engineer designs and maintains A data engineer designs and maintains an organization's data infrastructure.

[00:42:32]He designs and maintains an organization's entire data infrastructure.

[00:42:48]Data scientist strategizes for the future. Data scientist strategizes for the future.

[00:42:56]Data analyst summarizes and visualizes the data of the past.

[00:43:03]See, data scientist, right? He has got lot of other responsibilities like performing data analysis, data predictions, data classifications, clusterings, recommendations for the future. Data analyst summarizing, visualizing the data of the past. Data engineer, mainly for data management, data storage, and data processing. Data engineering, right? Data engineer should be working with all varieties of data.

[00:43:31]He should work with structured data. He should work with semi-structured. He should work with unstructured.

[00:43:36]He should work with streaming data. He should work with no sequel data.

[00:43:41]With all file formats like CSV, JSON, Parquet, XML, right? Yes.

[00:43:51]Okay.

[00:43:55]GCP, coming to this GCP Cloud, right?

[00:43:58]Say, computing service.

[00:44:01]It is allows customers to have platform as a service, infrastructure as a service, serverless computes.

[00:44:08]So, generally data engineer should state of quality.

[00:44:12]Optimizing data storage and data processing.

[00:44:17]Collaborating with the data scientist and data analyst to to provide as per the requirements, right?

[00:44:26]Monitoring data infrastructure.

[00:44:28]Monitoring that data is available at each phase of this process. ETL, extracting, transforming, and loading.

[00:44:35]This is the key task of a data engineer.

[00:44:38]Where he's [snorts] going to pull the data from different sources.

[00:44:41]And he was going to transform and load it.

[00:44:43]Various GCP products or services.

[00:44:58]Various GCP products or services.

[00:45:10]Mhm. [snorts] So, I was talking about various GCP products or services that I'll be discussing as part of this course.

[00:45:17]Go- Google Cloud Storage for storing the data.

[00:45:21]BigQuery for fast querying, storage, and analysis of large data sets.

[00:45:30]>> [clears throat] [snorts] >> Fast querying, storage, and analysis of large data sets.

[00:45:35]DataProc for big data processing and analytics.

[00:45:40]And Databricks PySpark processing in GCP.

[00:45:44]PySpark, very high-speed processing execution model for processing huge data at very quick time, right? Yes. Dataflow for streaming data which keeps on generating. Data which keeps such data we call it as streaming data.

[00:45:58]And batch data.

[00:46:19]And batch and batch data processing.

[00:46:26]B- Okay. Batch data means batch processing means millions and trillions of, right? User non-interactive applications.

[00:46:35]Batch applications, we say.

[00:46:39]Online processing, batch processing.

[00:46:41]Online processing means user interactive.

[00:46:44]Batch means user non-interactive.

[00:46:47]Any online application, user compulsory interacts by providing the details.

[00:46:52]But batch, there won't be any user involvement. There won't be any input of the user.

[00:47:00]Cloud Composer, Apache Airflow for automating and monitoring data pipelines.

[00:47:08]Pub/Sub for messaging and for data ingestions.

[00:47:13]Okay. So, here if I talk about this mainly like PySpark execution. PySpark is 100 times faster.

[00:47:23]PySpark is 100 times faster than normal disk processing.

[00:47:27]Normal disk processing, right? So, there was a test conducted, right? There was a test done.

[00:47:32]Yahoo has taken a table of 100 TB consisting of 1,024 columns.

[00:47:40]Task, sorting based on 16 columns.

[00:47:42]Sorting based on 16 columns.

[00:47:46]Time taken by Oracle has taken 3.5 days.

[00:47:52]MySQL, it has taken around 6 days.

[00:47:55]Teradata, it has taken 4.5 hours to process this 100 TB data, right? So, Teradata, Netezza has taken just 3 hours of time.

[00:48:07]See, this Teradata Netezza specialized database is used for data warehouse.

[00:48:12]Hadoop, big data Hadoop has just taken 3.4 minutes.

[00:48:17]Means as compared with this where they have taken days of time, this have taken hours of time.

[00:48:23]But it has taken minutes of time.

[00:48:26]With Spark Spark, PySpark, right?

[00:48:31]It's 100 times faster.

[00:48:33]Already Hadoop has got very high speed.

[00:48:36]So, Spark is 100 times faster than Hadoop.

[00:48:46]Yes.

[00:48:48]So, if I see, if you want to interact with the GCP services, if you want to interact, just I'll give some introduction to this. How we can interact with the GCP services? Many are there. GCP products I was talking about this GCP products, right? This one.

[00:49:03]Cloud Storage, Cloud SQL, BigQuery, Cloud DataProc.

[00:49:09]Mhm.

[00:49:10]All these things. [clears throat] If you want to interact with this, we have [snorts] got interacting interact with GCP services. We got majorly four methods.

[00:49:24]>> [snorts] >> But okay.

[00:49:41]>> Interacting with GCP services.

[00:49:48]Majorly three methods are there.

[00:49:51]Fourth method also available, but I'll discuss it separately.

[00:49:57]First one, using the console, >> [snorts] >> using the cloud shell, software development kit, SDK.

[00:50:28]Using the console, using cloud shell, using cloud SDK.

[00:50:47]>> [clears throat] >> Using console, using console if I talk, using console, right?

[00:50:53]A simple way.

[00:50:59]Just if you observe here, just I'll show you.

[00:51:04]Yes.

[00:51:12]>> [snorts] >> So, here >> [snorts] >> console.cloud.google.com.

[00:51:47]So, select that any One second.

[00:52:48]Okay, password.

[00:52:52]See, console.cloud.google.com is this.

[00:52:56]To open this uh console.cloud.google.com.

[00:53:00]Whenever you are taking a free account, right? So, it is going to provide you free accounts.

[00:53:06]Some of services are free for life long, some are free for 1 year, some are free for 3 months.

[00:53:14]So, yes, I'm great.

[00:53:27]So, this is cloud console.

[00:53:30]This what welcome to this cloud console, right? Yes.

[00:53:37]Okay.

[00:53:39]This is the console. If you This is what console where you want to open anything from here. You want to see this navigation menu where you can see anything here.

[00:53:48]All these components.

[00:53:52]This is what the project area.

[00:53:54]This what If you want to For example, cloud storage.

[00:54:00]Cloud storage if you see.

[00:54:04]Mhm.

[00:54:30]Now, can you see buckets?

[00:54:32]Any buckets available you can see.

[00:54:36]Many buckets are available.

[00:54:39]These are the various buckets that we created already.

[00:54:42]So, yes, if you want to go with cloud storage or if you want to check for cloud SQL, anything if you want to open, you can open from this console only. Any service, any product or any component, right? Cloud SQL.

[00:54:54]If you want to see cloud SQL from here itself, you can open cloud SQL.

[00:55:00]BigQuery.

[00:55:02]Anything.

[00:55:04]My SQL instance, post grace instance which I created.

[00:55:10]Yes.

[00:55:14]So, understand here.

[00:55:16]Using the console, using the console.

[00:55:22]This is what I was talking about console. Console is what? What is your understanding here?

[00:55:27]Console is a web-based a web-based You're not insta- Okay, it's a web-based graphical user interface, interface.

[00:55:40]A web-based UI.

[00:55:47]Using console, >> [snorts] >> we interact with GCP services.

[00:56:01]Okay.

[00:56:11]Here Here in console, we create a resource.

[00:56:17]We create a resource.

[00:56:20]You can delete a resource.

[00:56:27]We delete a resource.

[00:56:31]We update a resource.

[00:56:39]We update a resource.

[00:56:50]Apply or revoke permissions.

[00:56:54]Using console, you can perform all this.

[00:57:00]Okay, I'll come back to the cloud shell, but the difference between the difference between a service resource.

[00:57:14]What do you mean by service and resource?

[00:57:17]Service A simple way.

[00:57:21]Service resource.

[00:57:25]>> [cough] >> Service is offered by GCP.

[00:57:41]A resource created by user.

[00:57:50]So, service is offered by GCP, resource not like that.

[00:58:08]Service is offered by GCP, resource created by user.

[00:58:14]We have many services. We have many services.

[00:58:23]Example, BigQuery is there.

[00:58:28]BigQuery is a service.

[00:58:31]Under this, we create a resource.

[00:58:34]Under this, we create a we create resources.

[00:58:53]Like table one, table two, so on.

[00:58:59]>> [snorts] >> GCS.

[00:59:13]Under this, GCS means Google Cloud Storage.

[00:59:25]Like bucket one, bucket two, bucket three.

[00:59:39]>> So on. So in this way, service is offered by GCP, resource is created by the user.

[01:00:01]Okay, in this way many are there.

[01:00:04]Under this service, we create resources.

[01:00:07]Next.

[01:00:13]>> [snorts] >> Offered by GCP.

[01:00:25]For their own business purpose.

[01:00:44]But resources are resources you are creating.

[01:00:52]You are creating, right?

[01:01:01]For your own project purpose.

[01:01:05]For your own project purpose.

[01:01:10]Like tables, buckets, all this you are creating for your own project purpose.

[01:01:18]Okay.

[01:01:34]Next is Cloud Shell.

[01:01:38]The second purpose, second way is what?

[01:01:39]Cloud Shell. Up to now what? Cloud Console.

[01:01:42]Means just uh you can interact. Next is using Cloud Shell.

[01:02:00]It is a web-based command-line interface.

[01:02:05]It is a web-based command-line interface.

[01:02:09]Previous one, console, it is a web-based GUI, but it is web-based command-line interface.

[01:02:25]Here we use commands.

[01:02:57]>> [snorts] >> Here we use commands. Okay.

[01:03:03]>> [snorts] >> Okay, for example, how to work with Cloud Shell? How to work with Cloud Shell? Just observe how to work with Cloud Shell.

[01:03:15]Just observe. If I click on this >> [snorts] >> Cloud Google Cloud.

[01:03:24]This is the home page.

[01:03:27]Here you can see activate Cloud Shell. Activate Cloud Shell.

[01:03:56]Terminal.

[01:04:01]This is what? Whenever you say activate Cloud Shell, this is going to be open.

[01:04:05]Here you can work with anything. Here you can work with the command-based. This is command-based.

[01:04:15]BigQuery LS I'm saying. BigQuery LS I'm saying.

[01:04:20]In the BigQuery LS it is if it is got anything it will display, otherwise empty it will show. Empty it will show if there is nothing.

[01:04:42]Mhm, take some time.

[01:05:01]Mhm.

[01:05:02]No, nothing is there in that. Now, for example, GSUtil GSUtil list I'm saying. GSUtil list.

[01:05:17]It will list all the buckets.

[01:05:19]Google storage utility like list. It will display all the buckets.

[01:05:23]See, multiple buckets are available. I already created it.

[01:05:28]These are all the buckets. Buckets to display all the buckets.

[01:05:41]Here we are using commands to interact with GCP services.

[01:05:44]LS command, RM command, Linux-based commands we'll be using.

[01:05:50]Here if you want you can check it. You can work with Python.

[01:05:53]Python you can say enter.

[01:05:55]Python Shell 3.12 will be open. If you want to work with any valid Python statements.

[01:06:01]23 + 42.

[01:06:07]Gives the result.

[01:06:10]>> [snorts] >> Control L.

[01:06:12]Any valid Python statements. X as 10, Y as 4.

[01:06:16]What is X I'm asking? What is Y? What is X + Y?

[01:06:20]What is X - Y? X into Y?

[01:06:24]X by Y?

[01:06:26]Yes.

[01:06:27]X into 2.

[01:06:29]X squared, 10 squared, any valid Sorry, X squared.

[01:06:34]X squared.

[01:06:37]10 squared, 100.

[01:06:39]Control L.

[01:06:42]Mhm.

[01:06:43]>> [snorts] >> Okay.

[01:06:57]>> [clears throat] >> If you want to come out of this, say quit. Otherwise, if you want X equal to hello.

[01:07:03]X into 3 prints hello for three times.

[01:07:06]Any valid statements you can work on this Python statements.

[01:07:10]If you say Python {hyphen} {hyphen} version, it will display the Python version.

[01:07:18]Mhm.

[01:07:19]Java version.

[01:07:21]Java {hyphen} {hyphen} version.

[01:07:25]Displays Everything are available.

[01:07:27]Everything are Everything are already available.

[01:07:31]You no need to install these things.

[01:07:34]So here everything are available.

[01:07:36]Python Yes, we can create buckets from the shell also.

[01:07:47]You can create create from the buckets in the shell.

[01:07:50]You can check it in the console. You can create in the console. You can check it in this shell. Also, you can create from the Cloud SDK also. It is where you will be installing in your local machine.

[01:08:01]In your In your Windows system, you'll be installing the Cloud SDK. From there also you can create.

[01:08:07]>> [snorts] >> From there also you can create and you can create a buckets. You can check from there.

[01:08:13]And also you can go after creating. I'll create one bucket from console. I'll create one bucket from console. I can create one from shell. I can create one from SDK. Yes, multiple buckets I can create, but everything you can check it.

[01:08:28]You'll be able to view it.

[01:08:32]Yes. Okay, so sir So the most recommended way is to use console.

[01:08:38]Console will be like GUI-based it will be, but command command way also it will be very simple only single command.

[01:08:46]So creating bucket directly like create option will be there. If you click on that, it will ask for the name. You will give some name and say create. It gets created.

[01:08:56]Command also it will be very simple creating bucket in single line.

[01:09:01]Anyway anyway, depending upon that, right? Programmatically if you want to go, you go for this command-based.

[01:09:09]Okay.

[01:09:13]Fine.

[01:09:14]So that way what I'm saying about how we can interact with the using a command-based you can, otherwise GUI-based console. GUI-based console or command-based CLI. You can use any of them. Everything are available here. And again, if we if we talk about this CLI again, this it is of two types.

[01:09:37]If you observe.

[01:09:42]Cloud shell and Cloud shell terminal and editor. Terminal and editor, two are there. Now, this is terminal.

[01:09:51]You want to say open editor, you can also open editor editor.

[01:09:56]Open editor.

[01:10:00]This is an editor.

[01:10:03]Just like VS Code it will be it will be just like VS Code.

[01:10:09]You know, see if you have idea about VS Code, this like VS Code.

[01:10:13]If you want to create any file and if you want to execute here I can right click.

[01:10:21]I can just click on this.

[01:10:24]Want to create a file, I'll create a file.

[01:10:36]123.py demo.py I'm saying 123.py just X as 10 Y as 20 print X + Y. I can type here. I can write some code within this.py file.

[01:10:55]I can execute it.

[01:11:03]>> [laughter] >> Next So sir, basically this GCP is you know, it is like a one place destination where you can code you can deploy and you can run it. Is it like that?

[01:11:27]Yes, GCP everything you can perform here within the cloud environment.

[01:11:33]Okay, even the coding?

[01:11:34]Yes, everything you can do it.

[01:11:38]Okay.

[01:11:39]So if I want to push my code, where I can push my code in GCP?

[01:11:44]I'll discuss that.

[01:11:45]One second.

[01:11:48]So here, we have got many things, right?

[01:11:50]Many things, many components, many services. If you want to work with the things, right? One by one I'll be discussing.

[01:11:56]So here I was talking about this console shell, right? Where you can interact with any services here from if you want to interact with the Cloud SQL or Cloud Storage or BigQuery, anything, right? You can just Okay.

[01:12:13]Now See the difference between these two.

[01:12:17]See the difference between these two.

[01:12:20]It is a cloud-based CLI.

[01:12:23]Used commands to interact with the services.

[01:12:26]Even SDK also, the next one.

[01:12:30]The third one Cloud SDK also it is command-based only. But what is the difference between Cloud Shell and Cloud SDK?

[01:12:38]Cloud Shell and Cloud SDK, right?

[01:12:43]Yes.

[01:12:45]In Cloud Shell, it is web-based.

[01:12:48]Cloud SDK is not web-based.

[01:12:50]SDK is not web-based. It is local-based. In your local machine you will install and you will work with it.

[01:13:00]Just what is it?

[01:13:03]You will just install in your local machine and you will work with it. It is local-based. It's not web-based. But Cloud Shell and Cloud console, both are web-based. But this is not web-based. It is local. We'll be installing. But what all the commands we are using in this Cloud Shell for interacting with the various services the command same commands you can use to work with Cloud SDK.

[01:13:24]It is also command-based. Here also we used commands for interacting with the services.

[01:13:29]Here also we'll be using commands to interact with the services.

[01:13:33]So this are the interacting Cloud SDK, Cloud Shell. Cloud SDK we have I'll be I'll be showing you step by step how to download and how to install and how to work on it.

[01:13:44]More on this. But the Cloud Shell, Cloud SDK, just I was talking about shell.

[01:13:49]This command line interface where you can write and work with the commands.

[01:13:54]Yes. So once you can check with that, right? What I'm going to discuss as part of it.

[01:13:59]See, GCP data engineering fundamentals, Google Cloud Storage Cloud SQL setting up database, BigQuery for building data warehouse.

[01:14:09]One second.

[01:14:13]DataProc for big data processing, DataBricks for PySpark processing, DataFlow for Apache Beam development Google Cloud Composer for orchestration DataFusion for data integration DataBuildTool, Airflow and Terraform will be discussing. These 10 things, right? In brief we'll be discussing.

[01:14:36]Introduction to this Cloud Platform, GCP data engineering fundamentals, starting with GCP, getting a free account, billing, project creation storage and databases in GCP, working with the Google Cloud Storage, setting up a data lake using GCS Cloud SQL setting up Postgres database using Cloud SQL data integrations and data ingestions data processing and transformations data warehousing and analytics using like BigQuery BigQuery for building data warehouse in brief we'll be discussing, right?

[01:15:13]Yes, DataProc for big data processing.

[01:15:17]DataProc for big data processing and see ETL data pipelines using DataProc.

[01:15:24]And PySpark processing in GCP.

[01:15:29]ELT pipeline using DataBricks.

[01:15:33]Integration of Spark on DataProc and BigQuery.

[01:15:38]DataFlow Apache Beam development.

[01:15:43]Batch and full streaming process.

[01:15:45]Pub/Sub Google Cloud Composer for data pipeline orchestration.

[01:15:53]DataFusion Cloud Functions Terraform Data pipelines using DB2, Airflow and BigQuery.

[01:16:07]Machine learning in GCP.

[01:16:09]Real world GCP data engineering scenarios, building streaming pipeline building a batch pipeline, data migrations on-prem to GCP.

[01:16:18]So if you want to migrate from on-premises to GCP, designing a hybrid data lake BigQuery data lakes in GCS, right? Yes.

[01:16:28]So this what in brief in depth we'll be discussing, right? You can just check with this offerings from my side.

[01:16:37]Each and every session video you will be getting, soft copy of the class notes assignments and tasks to work with, a WhatsApp group where you can directly interact with me.

[01:16:47]Interview questions around 200 live interactive hands-on sessions Python videos, my And from DurgaSoft site they'll be the prerequisites like basic Python, like basic SQL, basic Linux, right?

[01:17:01]So for that it is providing 100 videos on Python. You can just check with that.

[01:17:05]Some 30 or 40 videos on this MySQL.

[01:17:09]Some 100 videos on Linux, right? You can just check with that.

[01:17:12]It is providing free access to this videos, MySQL, Linux and Python once you get enrolled to this course.

[01:17:19]So everything from the prerequisites, everything I'll be discussing.

[01:17:22]So from the next week we'll be starting with the actual things.

[01:17:27]The getting the subscription, working with the practical part.

[01:17:31]So next week the link will be changing.

[01:17:32]You are supposed to get enrolled to get the new link.

[01:17:36]You can see in the Zoom chat a big message is given by this online team.

[01:17:40]You can work with that.

[01:17:42]You can just check. You can just copy that big message, right?

[01:17:50]I don't see any message, sir.

[01:17:56]One second.

[01:18:36]Okay. So the duration of the one they'll be posting now.

[01:18:40]So the duration of this course will be 12 weekends, 3 months.

[01:18:46]So 2 hours on Saturday, 2 hours on Sunday.

[01:18:49]Initially for the first 3 4 weeks still be at 5:00 to 7:00.

[01:18:57]Later it will be 6:00 p.m. to 8:00 p.m.

[01:19:00]So like 2 hours in the evening Saturday evening and 2 hours in the Sunday evening.

[01:19:06]It goes like around 12 weekends it goes.

[01:19:08]We'll be discussing about all this practically.

[01:19:11]Those who wants to do the things practical practical things if you want to do parallel along with me during the session, you can do it.

[01:19:18]If any errors in the settings or if you want to creating anything, right?

[01:19:22]You can just share the screens. And still if any errors while practicing, no need to wait until the next week.

[01:19:31]You can just WhatsApp group is available. You can post that errors. You can take snaps of it and post in the WhatsApp group. They'll be handled.

[01:19:37]They'll be answered, right? Yes.

[01:19:42]So, those who doesn't have anyhow, I'll be discussing the Python syntaxes, Linux syntaxes, right? Before I go. Not But still if you want to have to have command or by the next week we'll be starting with this one week if you are free, just if you doesn't have knowledge, you can just go with that.

[01:19:58]Get enrolled and get the videos of this Python, MySQL, and Linux.

[01:20:02]And go through those if you doesn't have knowledge of those, right?

[01:20:06]Yes. Now, Okay.

[01:20:12]So, you will get a mail, right? You will get a mail about this uh uh course content details, payment details, all this. Get enrolled for getting this video access.

[01:20:23]You'll be getting clear notes.

[01:20:26]Yes. Next week, the link will be changing. You are supposed to get enrolled for getting the paid link, right? To your mail.

[01:20:33]Daily daily videos will be added to your Google Drive. Notes you'll be getting to your mail.

[01:20:39]Yes.

[01:20:43]Okay.

[01:20:44]So, we'll start from the next week with the actual things, right? Yes. So, they'll forward actually they didn't post here.

[01:20:50]Second.

[01:20:52]They'll post they'll provide you through mail, right? With whatever mail you have joined the meeting to that mail you'll be getting, right? The details about the payment. They'll There is a note given in that Okay, you can just check with that all the whatever the things.

[01:21:08]Fine.

[01:21:10]So, getting this a free subscription for 3 months.

[01:21:14]Within the 3 months you can work with freely. Don't take any paid subscription.

[01:21:18]Free subscription is more than enough.

[01:21:20]26,000 credits will be added.

[01:21:23]With the 26,000 credits you can just use all that products, premium products also, services also can be used.

[01:21:30]Okay.

[01:21:32]Okay, fine. So, we'll meet on next Saturday. More discussion on this.

[01:21:37]We'll continue the course from there, right?

[01:21:40]Thank you for your time. Thank you. Meet you on next Saturday.

[01:39:14]>> Mhm.

[01:49:09]>> Mhm.

[01:59:04]>> Mhm.

[02:08:59]>> Mhm.

[02:18:54]>> Mhm.

[02:28:49]>> Mhm.

[02:38:44]>> Mhm.

[02:48:39]>> Mhm.

[02:58:34]>> Mhm.

[03:00:29]>> Mhm.