The Role:

TripleLift is seeking a Senior Data Engineer to join a small, influential Data Engineering team. You will be responsible for expanding and optimizing our high-volume, low-latency data platform architecture, as well as optimizing data flow and collection for cross functional teams. The ideal candidate is an experienced data pipeline engineer and data wrangler who enjoys optimizing data systems and building them from the ground up. This role will support our software engineers, product managers, business intelligence analysts and data scientists on data initiatives, and will ensure optimal data delivery architecture is applied consistently throughout new and ongoing projects. Ideal candidates must be self-directed and comfortable supporting the data needs of multiple teams, systems and products. The right candidate will be excited by the prospect of optimizing or even re-designing our company’s data architecture to support our next generation of products and data initiatives.

Responsibilities:

• Create and maintain optimal, high-throughput data platform architecture handling 100’s of billions of daily events.
• Explore, refine and assemble large, complex data sets that meet functional product and business requirements.
• Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
• Build the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using Spark, EMR, Snowpark, Kafka and other big data technologies.
• Work with stakeholders across geo-distributed teams, including product managers, engineers and analysts to assist with data-related technical issues and support their data infrastructure needs.
• Digest and communicate business requirements effectively to both technical and non-technical audiences.
• Translate business requirements into concise technical specifications.

Qualifications:

• 6+ years of experience in a Data Engineer role
• Bachelors Degree, or higher, in Computer Science or related Engineering field
• Experience building and optimizing ‘big data’ data pipelines, architectures and data sets
• Expert working knowledge of Databricks/Spark and associated APIs
• Strong experience with object-oriented and functional scripting languages: Python, Java, Scala and associated toolchain
• Experience working with relational databases, SQL authoring/optimizing as well as operational familiarity with a variety of databases.
• Experience with AWS cloud services: EC2, EMR, RDS
• Experience working with NoSQL data stores such as: Elasticsearch, Apache Druid
• Experience with data pipeline and workflow management tools: Airflow
• Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
• Strong experience working with unstructured and simi-structured data formats: JSON, Parquet, Iceberg, Avro, Protobuf
• Expert knowledge of processes supporting data transformation, data structures, metadata, dependency and workload management.
• Proven experience in manipulating, processing, and extracting value from large, disparate datasets.
• Working knowledge of streams processing, message queuing, and highly scalable ‘big data’ data stores.
• Experience supporting and working with cross-functional teams in a dynamic environment.

Preferred: 

• Streaming systems experience with Kafka, Spark Streaming, Kafka Streams
• Snowflake/Snowpark
• DBT
• Exposure to AdTech