What you will be doing:

We are looking for a highly skilled Data Infrastructure Engineer to join our team. The Data Infrastructure Engineer will be responsible for the management, monitoring, backup, cost management, performance tuning, and deployment of a variety of highly available and fault-tolerant SQL, NoSQL, Kafka, and Spark-based data layer systems running on AWS and GCP. The ideal candidate will have experience working with various data processing tools and cloud platforms.

As a Data Infrastructure Engineer, you will

• Design, implement, and maintain data layer orchestration systems to expose and manage a series of self-managed and PaaS data layer technologies
• Manage and monitor data layer components and services on AWS and GCP, ensuring optimal performance and availability
• Configure and deploy backup solutions for databases, Kafka, and Spark, ensuring data integrity and facilitating disaster recovery
• Implement cost management strategies for database and data processing infrastructure, optimising resource utilisation and minimising expenses
• Perform regular performance tuning for databases, Kafka, and Spark, identifying optimization opportunities and implementing improvements. Troubleshoot and resolve issues related to these systems
• Create and maintain documentation for Data Infrastructure engineering processes, infrastructure, and applications
• Work closely with other teams to integrate data sources and ensure data consistency, and design and model databases that support application requirements, and adhere to best practices for normalisation and performance.
• Monitor and manage the capacity of data infrastructure, planning for future growth and scalability; analyse query performance and implement appropriate indexing strategies
• Provide technical support and guidance to other team members in relation to Data Infrastructure engineering tasks and technologies
• Collaborate with stakeholders to define and prioritise Data Infrastructure engineering projects, setting realistic timelines and expectations
• Collaborate with Software Engineers to create resilient data sharding strategies 

Our Tech Stack:

ComplyAdvantage is fully cloud-based, with a modern kubernetes-focused tech stack. All compute workloads run in Kubernetes, with clusters in multiple regions to support the needs of our global client base. Our production services are currently based in AWS and GCP.

We make heavy use of Terraform and Helm to define our infrastructure and services, and lean heavily on GitOps paradigms - production and non-production environments are defined in git and changes to these environments (both cloud infrastructure and Kubernetes applications) are managed via git.

Kafka is used widely across teams, and is seen as a core technology in our tech stack. We are focusing on developments to our Kafka and Yugabyte tooling, as well as the other data systems we rely on, such as PostgreSQL, Elasticsearch and Databricks/Spark. To tie in with our multi-cloud plans, we’ll also be developing inter-cloud failover options to be leveraged by our development teams.

Across the wider Technology team, teams build and release containerised applications to support the wide array of activities that our teams are engaged in - from developing low latency client-facing APIs, to machine learning models and data processing pipelines.

About you:

As an experienced Data Infrastructure Engineer, you will have

• Significant experience working with AWS and/or GCP core services
• Significant experience building, scaling and tuning data systems such as Apache Kafka, Apache Spark, Elasticsearch and PostgreSQL-compatible databases
• An awareness of cost and performance considerations when designing data systems architectures
• Some experience working with Kubernetes from a cluster admin perspective
• Experience codifying infrastructure as code in Terraform and/or Helm charts
• Experience working with observability tooling (logging, monitoring, tracing)
• Experience with CI tooling, including test pipelines and asset generation and publishing (eg Docker images, helm charts)
• Awareness of security considerations
• Experience with data modelling, sharding and indexing strategies running databases at scale

Nice to have:

• Experience running databases within Kubernetes
• Experience of using build-test-run CI/CD techniques to IaC.
• Experience of distributed SQL and/or noSQL databases, such as Yugabyte, Cockroach, Spanner, HBase or CouchDB

What’s in it for you? 

• Equity as we want you to have a part of what we are building 
• Unlimited Time Off Policy- A work-life balance and focus on our well-being are critical to keeping us performing at our best 
• We embrace a hybrid approach that requires employees to be in the office for two days a week. We strongly believe that this approach fosters collaboration and enables the building of meaningful relationships
• You will also get a new starter budget of €235 to kit out your home office 
• Opportunity to work on innovative projects with smart-minded people keen to share their knowledge and continuously improve 
• Annual learning budget of €2200 (prorated based on start date) to drive your performance and career development