Design, develop, test, and deploy high-performance Apache Spark/Scala data processing applications and ETL pipelines on Cloudera (CDH). Optimize Spark and platform performance, build data pipelines using HDFS, Hive, Impala, HBase, and Kafka, ensure data integrity and security, collaborate with data scientists and analysts, and implement version control and CI/CD for deployments.
Key Responsibilities:
- Develop, test, and deploy data processing applications using Apache Spark and Scala.
- Optimize and tune Spark applications for better performance on large-scale data sets.
- Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase, Kafka) to build data pipelines and storage solutions.
- Collaborate with data scientists, business analysts, and other developers to understand data requirements and deliver solutions.
- Design and implement high-performance data processing and analytics solutions.
- Ensure data integrity, accuracy, and security across all processing tasks.
- Troubleshoot and resolve performance issues in Spark, Cloudera, and related technologies.
- Implement version control and CI/CD pipelines for Spark applications.
Required Skills & Experience:
- Minimum 8 years of experience in application development.
- Strong hands on experience in Apache Spark, Scala, and Spark SQL for distributed data processing.
- Hands-on experience with Cloudera Hadoop (CDH) components such as HDFS, Hive, Impala, HBase, Kafka, and Sqoop.
- Familiarity with other Big Data technologies, including Apache Kafka, Flume, Oozie, and Nifi.
- Experience building and optimizing ETL pipelines using Spark and working with structured and unstructured data.
- Experience with SQL and NoSQL databases such as HBase, Hive, and PostgreSQL.
- Knowledge of data warehousing concepts, dimensional modeling, and data lakes.
- Ability to troubleshoot and optimize Spark and Cloudera platform performance.
- Familiarity with version control tools like Git and CI/CD tools (e.g., Jenkins, GitLab).
Similar Jobs
Cloud • Security • Software • Cybersecurity • Automation
Full lifecycle recruiter supporting AMER sales roles: sourcing passive and active candidates, partnering with hiring teams, using ATS and sourcing tools, leveraging market data and metrics to improve pipelines and candidate experience during a six-month contract.
Top Skills:
GitGitlabGreenhouseLinkedin Recruiter
Marketing Tech • Real Estate • Software • PropTech • SEO
Build and own the analytics foundation: maintain dbt models and Snowflake, create ELT pipelines (Python/Airflow), design semantic layers, ensure data quality and reconciliation, enable self-serve analytics and measurement for cross-functional teams, and support AI-ready data infrastructure.
Top Skills:
AirflowAirflow DagsApi IntegrationsBigQueryCi/CdDbtDbt Semantic LayerEltGitMixpanelPosthogPythonRedshiftSalesforceSnowflakeSnowflake CortexSQL
Big Data • Fintech • Mobile • Payments • Financial Services
Lead technical strategy and roadmap for Affirm's Online Storage platform. Design and build scalable, multi-region datastore solutions, automation control planes, and operational tooling for hundreds of databases. Drive cross-team collaboration, reliability, monitoring, and mentor engineers while delivering high-impact backend and database features.
Top Skills:
AWSDistributed SqlDynamoDBKotlinKubernetesMySQLPgbouncerPostgresProxysqlPythonRds ProxyRedisSparkTidbVitess
What you need to know about the Montreal Tech Scene
With roots dating back to 1642, Montreal is often recognized for its French-inspired architecture and cobblestone streets lined with traditional shops and cafés. But what truly sets the city apart is how it blends its rich tradition with a modern edge, reflected in its evolving skyline and fast-growing tech industry. According to economic promotion agency Montréal International, the city ranks among the top in North America to invest in artificial intelligence, making it le spot idéal for job seekers who want the best of both worlds.
Key Facts About Montreal Tech
- Number of Tech Workers: 255,000+ (2024, Tourisme Montréal)
- Major Tech Employers: SAP, Google, Microsoft, Cisco
- Key Industries: Artificial intelligence, machine learning, cybersecurity, cloud computing, web development
- Funding Landscape: $1.47 billion in venture capital funding in 2024 (BetaKit)
- Notable Investors: CIBC Innovation Banking, BDC Capital, Investissement Québec, Fonds de solidarité FTQ
- Research Centers and Universities: McGill University, Université de Montréal, Concordia University, Mila Quebec, ÉTS Montréal
