Sr. Consulting / Principal Software Engineer

Are you a proactive team player?

Do you want to drive growth and advise internal stakeholders as a strategy professional?

About Team

The Life Sciences Solutions (LSS) platform at Elsevier is an initiative focused on enhancing the company's digital presence and improving customer engagement. The platform includes the development of a new website, the implementation of a customer relationship management (CRM) system, and the creation of a mobile application.

About Role

We are seeking an Experienced Data Platform Engineer with expertise in Scala, Java, semantic technologies, and knowledge representation. In this senior role, you will collaborate with architects, data scientists, and engineers to design and enhance our data platform for both unstructured and structured data.

Your responsibilities include:

• Designing scalable data processing architectures
• Working with structured and unstructured data
• Leveraging cloud technologies and microservices
• Applying software engineering best practices
• Collaborating with data scientists to operationalize models
• Ensuring data quality and consistency

Candidates who utilize functional programming principles to create reliable, maintainable, and scalable solutions are highly valued. If you're eager to advance our data platform's efficiency, reliability, and scalability, we'd like to hear from you. 

Responsibilities

Data Platform Architecture and Evolution:

• Collaborate to optimize the data platform for scalability and efficiency, aligning with business goals.
• Apply principles like hexagonal, medallion, and event-driven architectures.
• Transition from data lake to data lakehouse using Delta Lake on Databricks.
• Partner with the Team Delegate Architect to refine the technology stack.

Implement and Enhance Scalable Data Platform Components:

• Develop and maintain scalable data processing microservices in Kubernetes or Databricks using Scala, Java, fs2, Apache Spark, Kafka, SQS, and S3.
• Apply functional programming principles with the Type level ecosystem (e.g., Cats, Cats Effect) for concurrency management, complex data processing, and type-safe code.
• Use stream processing technologies like Kafka Streams or Spark Structured Streaming with Delta table for incremental data updates.
• Employ advanced messaging patterns, such as the claim check pattern, for asynchronous large message handling within microservices.

Data Processing Flow:

• Implement efficient batch and streaming data processing within the platform.
• Develop monitoring and alerting systems to ensure data flow reliability.
• Regularly analyze, monitor, and optimize data flow performance and platform scalability, resolving bottlenecks.

Structured and Unstructured Data Processing:

• Develop microservices to extract and enrich unstructured documents, using web annotation ontologies.
• Combine extracted content with structured data to create reusable datasets like Knowledge Graphs.
• Ensure seamless integration of both data types, applying SHACL validation for quality and consistency. 

Collaborate with Data Scientists:

• Work closely with data scientists to incorporate their models into the data platform processing flows, ensuring scalability and performance. 

Mentorship and Collaboration:

• Guide team members technically across various tech stacks.
• Foster a collaborative environment valuing diverse perspectives.
• Align with the team architect on beneficial architectural decisions.
• Promote continuous learning, knowledge sharing, and staying current with industry trends. 

Requirements:

• 10+ of Software Engineering Experience. 
• BS Engineering/Computer Science or equivalent experience required; advanced degree preferred. 
• Expertise in Scala and Java, with a focus on functional programming for data engineering.

• Solid understanding of semantic technologies and knowledge representation (RDF, OWL, JSON-LD).
• Experience in data validation and quality assurance, including SHACL.
• Proficiency in data query languages (SPARQL, SQL) and transformation techniques (XSLT).
• Strong data modelling skills (relational and graph).
• Knowledge of various data storage formats and their implications.
• Familiarity with advanced data integration, such as Knowledge Graph Construction.
• Expert in scalable systems using microservices and event-driven architectures.
• Familiarity with hexagonal/onion architecture.
• Proficient in deploying microservices with Kubernetes and managing CI/CD pipelines.
• Extensive experience with AWS for data platforms.
• Experience with big data frameworks (Apache Spark, Kafka Streams).
• Knowledge of data Lakehouse architectures (Delta Lake, Apache Iceberg).
• Competent in monitoring tools and optimizing data platform performance.
• Strong problem-solving skills in cloud environments.
• Experienced in agile software development.
• Excellent communication skills and ability to collaborate across teams. 

Work in a way that works for you

We promote a healthy work/life balance across the organisation. We offer an appealing working prospect for our people. With numerous wellbeing initiatives, shared parental leave, study assistance and sabbaticals, we will help you meet your immediate responsibilities and your long-term goals.

• Working flexible hours - flexing the times when you work in the day to help you fit everything in and work when you are the most productive

Working for you

We know that your wellbeing and happiness are key to a long and successful career. These are some of the benefits we are delighted to offer:

• Generous holiday allowance with the option to buy additional days
• Health screening, eye care vouchers and private medical benefits
• Wellbeing programs
• Life assurance
• Access to a competitive contributory pension scheme
• Save As You Earn share option scheme
• Travel Season ticket loan
• Electric Vehicle Scheme
• Optional Dental Insurance
• Maternity, paternity and shared parental leave
• Employee Assistance Programme
• Access to emergency care for both the elderly and children
• RECARES days, giving you time to support the charities and causes that matter to you
• Access to employee resource groups with dedicated time to volunteer
• Access to extensive learning and development resources
• Access to employee discounts scheme via Perks at Work

About the Business

A global leader in information and analytics, we help researchers and healthcare professionals’ advance science and improve health outcomes for the benefit of society. Building on our publishing heritage, we combine quality information and vast data sets with analytics to support visionary science and research, health education and interactive learning, as well as exceptional healthcare and clinical practice. At Elsevier, your work contributes to the world’s grand challenges and a more sustainable future. We harness innovative technologies to support science and healthcare to partner for a better world.