Consultant Software Engineer - Elastic search administrator

Position Overview:

We are seeking an experienced Senior Kafka & NiFi Administrator to design, deploy, manage, and optimize enterprise‑scale data streaming and data flow platforms. The ideal candidate will have deep expertise in Apache Kafka, Kafka Connect, Schema Registry, and Apache NiFi, along with strong automation, scripting, CI/CD, and cloud-native experience. This role ensures high availability, secure configurations, performance tuning, and operational excellence for real-time data pipelines.

Key Responsibilities:

1.      Kafka Administration

a.      Install, configure, and manage Apache Kafka clusters (on‑premise or cloud-native: Azure HDInsights).

b.      Manage Kafka ecosystem components: Kafka Connect, Schema Registry, Kafka Streams, ZooKeeper, etc.

c.      Perform cluster scaling, partition rebalancing, topic management, multi‑DC replication (MirrorMaker).

d.      Implement monitoring, alerting, and logging using tools such as Prometheus, Grafana, ELK/ECE.

e.      Ensure Kafka security (TLS encryption, SASL, RBAC, Kerberos, OAuth)

f.        Troubleshoot broker issues, performance bottlenecks, consumer lag, and message serialization errors.

2.      NiFi Administration

a.      Install, manage, and maintain Apache NiFi, NiFi Registry

b.      Design, optimize, and troubleshoot complex NiFi data flows

c.      Manage NiFi cluster configuration, back-pressure settings, tuning, and provenance repository

d.      Integrate NiFi with Kafka, S3, HDFS, RDBMS, REST APIs, and cloud services

e.      Implement access control, SSL/TLS security, policies, and NiFi user/group management.

3.      Automation & DevOps

a.      Develop automation using Ansible, Terraform or Bash

b.      Build CI/CD pipelines for Kafka, NiFi, and data flow deployments using Jenkins, Azure DevOps, GitHub Actions.

c.      Automate cluster provisioning & configuration using Ansible & Terraform

d.      Create reusable templates and automation for topic creation, ACL management, connector deployment, and flow lifecycle.

4.      Operations & Support

a.      Provide L3 support for streaming platforms, including incident analysis and root cause identification.

b.      Establish and enforce best practices for data governance, data flow reliability, and operational standards

c.      Maintain detailed documentation for configurations, architectures, and runbooks

d.      Collaborate with platform engineering, data engineering, security, SRE, and cloud teams

Key skills required:

1.      Must have:

a.      6–10+ years of experience on Kafka & NiFi Administration

b.      Strong Knowledge of Bigdata Architecture & Administrator's role

c.      Strong Knowledge of Bigdata Hadoop, Kafka Internals, NiFi flow design, performance tuning, various data formats, Schema registry, Kafka Connect, etc.

d.      Configuration and Performance Tuning of Kafka & NiFi clusters

e.      Application Deployment and Disaster Recovery

f.        Automation on Bigdata Infra with the help of Ansible & Terraform

2.      Good to have:

a.      Java & Shell scripting

b.      Hadoop Administration

c.      Excellent communication skills

Must have:

a.      6–10+ years of experience on Kafka & NiFi Administration

b.      Strong Knowledge of Bigdata Architecture & Administrator's role

c.      Strong Knowledge of Bigdata Hadoop, Kafka Internals, NiFi flow design, performance tuning, various data formats, Schema registry, Kafka Connect, etc.

d.      Configuration and Performance Tuning of Kafka & NiFi clusters

e.      Application Deployment and Disaster Recovery

f.        Automation on Bigdata Infra with the help of Ansible & Terraform

2.      Good to have:

a.      Java & Shell scripting

b.      Hadoop Administration

c.      Excellent communication skills

We are committed to creating a diverse environment and are proud to be an equal opportunity employer. All qualified applicants receive consideration for employment without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, disability, age, or veteran status

At Société Générale, we are convinced that people are drivers of change, and that the world of tomorrow will be shaped by all their initiatives, from the smallest to the most ambitious. Whether you’re joining us for a period of months, years or your entire career, together we can have a positive impact on the future. Creating, daring, innovating, and taking action are part of our DNA. If you too want to be directly involved, grow in a stimulating and caring environment, feel useful on a daily basis and develop or strengthen your expertise, you will feel right at home with us!

Still hesitating?

You should know that our employees can dedicate several days per year to solidarity actions during their working hours, including sponsoring people struggling with their orientation or professional integration, participating in the financial education of young apprentices, and sharing their skills with charities. There are many ways to get involved.

We are committed to support accelerating our Group’s ESG strategy by implementing ESG principles in all our activities and policies. They are translated in our business activity (ESG assessment, reporting, project management or IT activities), our work environment and in our responsible practices for environment protection.