Data Engineer – Big Data
Bengaluru, Karnataka, India Remote
About the job
About us: Quantiphi is an applied AI and Data Science Software and Services Company driven by the desire to solve transformational problems at the heart of the business. We combine deep industry experience, disciplined cloud, and data engineering practices, and cutting-edge artificial intelligence research to achieve quantifiable business impact; making products smarter, customer experiences frictionless, and processes autonomously while uncovering anomalies, threats, and risks that make businesses safer.
For more exciting insider, check out our website: https://quantiphi.com
We have multiple openings across Lead and Sr (3-10 years)
- Design and build a hybrid data solution using a combination of on-premise and in-cloud services.
- Diagnose and troubleshoot complex distributed systems problems and develop solutions with a significant impact on our massive scale.
- Build tools to ingest and process terabyte-scale data on a daily basis.
- Communicate with a wide set of teams, including product owners, business analysts, data and enterprise architects, platform engineers, and cloud vendors, etc.
- Compilation of detailed design documents, technical specification documents, and test plans as required.
- Independently manage development tasks as per technical specification documents, prod deployment, post-prod support, etc
- Managing support to Unit Testing, SIT & UAT.
- Coordinate with business and support teams on testing and migration activities.
Roles & Responsibilities For Data Engineer
- Experts in SQL writing and performance optimization on BigQuery.
- Experience in data validation.
- Experience in large scale data migration projects – history loads, incremental loads.
- Ability to decipher the logic and identify the inputs and outputs associated with individual queries
- Ability to build wrapper scripts to facilitate execution of independent SQLs within BQ for functional testing and benchmarking.
- Execute individual BQ scripts/procedures to validate functionality and capture performance benchmark
- Experience in tuning & optimizing SQL queries within BQ.
- Strong troubleshooting skills to identify the root cause of performance bottlenecks.
- Effectively articulate technical issues to Quantiphi tech lead, Merkle team and facilitate resolution.
- Create scripts and perform data validation/consolidation across systems.
- Hadoop, Hive, Spark, PySpark, Python, SQL, Snowflake or AWS Redshift
To prepare for this job, check out tons of free content here