What is Data Engineering ?
There is no particular definition for Data Engineering or Data Engineer. If you googled you will encounter more than 1,00,000+ results. By putting all the info at a place, according to Fundamentals of Data Engineering (By Joe Reis and Matt Housley) the definition follows as :
“Data Engineering is development, implementation and maintaince of systems and processes that takes in raw data and produces high quality, consistent information that supports downstream usecases such as analysis and machine learning”
“Data Engineer manages the Data Engineering Life Cycle”
Data Engineering Life Cycle:
Data Engineering Life Cycle Includes various stages like
1. Generation
2. Storage
3. Ingestion
4. Transformation
5. Serving
It also includes some undercurrents like Security, Data Management, Data OPS, Data Architecture, Orchestration and Software Engineering.
**Orchestration refers to the coordination and management of various data processing tasks and workflows
