Software Engineer - Big Data Ingestion and Processing

1 Month ago • 5-7 Years

Job Summary

Job Description

As a Software Engineer, you will be instrumental in managing large datasets for a critical intelligence mission. Your key responsibilities include loading large datasets, developing ingestion algorithms, optimizing data ingest processes, and creating Apache NiFi schemas. You will develop software tools for preprocessing, modifying, and archiving data in near real-time. Additionally, you'll ensure proper access controls, generate metrics for data integrity, and document data flows. You'll collaborate with data scientists, analysts, and managers in a dynamic environment.

Must have:

Experience in Computer Science or related field.
Experience with AWS cloud services.
Experience working with Databricks.
Understanding of SQL database structures.
Experience working with Apache NiFi.
Experience with large data clusters.
Experience with API development techniques.
Experience developing ETL processes.
Experience creating OS scripts for ETL.
Experience with Git version control.
Experience testing software solutions.
Experience implementing multiprocessing data-flows.

9 skills required

9 skills required for this role

Add these skills to join the top 1% applicants for this job

data-analytics

github

quality-control

aws

hadoop

git

python

sql

algorithms

Job Details

About the Organization

Now is a great time to join Redhorse Corporation. Redhorse specializes in developing and implementing creative strategies and solutions with private, state, and federal customers in the areas of cultural and environmental resources services, climate and energy change, information technology, and intelligence services. We are hiring creative, motivated, and talented people with a passion for doing what's right, what's smart, and what works.

About the Role

Redhorse is transforming how government agencies leverage data and technology. We are seeking a highly skilled Software Engineer to join our team supporting a critical intelligence mission. You will play a vital role in ingesting, processing, and analyzing massive datasets, directly impacting the Sponsor's ability to address pressing intelligence questions. You will work with cutting-edge technologies in a dynamic, collaborative environment, directly contributing to national security.

Key Responsibilities

Load large datasets into the Sponsor’s on-premises and Cloud environments.
Develop and maintain ingestion algorithms and schemas for large datasets.
Analyze new large-volume datasets to optimize the data ingest processes.
Support the creation of Apache NiFi schemas for new data loads.
Develop software tools that efficiently preprocess, modify, aggregate, load, index, and archive large data collections into clusters in near real-time.
Ensure proper access controls are implemented.
Generate metrics to track data ingest statistics to maintain data integrity and provenance.
Document the data-flows according to standards set by the Sponsor.
Engage regularly with data scientists, analysts, and managers.

Required Experience/Clearance

Demonstrated professional experience in Computer Science, Computer Engineering, Systems Engineering, or closely related discipline.
Demonstrated professional experience with AWS cloud services, including long-term storage options, and cloud-based database services.
Demonstrated experience working with Databricks.
Demonstrated experience understanding SQL database structures and mapping them between different SQL databases.
Demonstrated professional experience working with Apache NiFi.
Demonstrated professional experience working with large data and high-performance compute clusters such as Hadoop or similar.
Demonstrated experience with API development techniques.
Demonstrated experience developing and deploying ETL processes for large data sets.
Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases.
Demonstrated professional experience with version control systems, preferably Git.
Demonstrated experience testing the development of software solutions for the extraction, transformation, and loading of data using the most efficient languages for the task such as NiFi, Python, and SQL.
Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations.
Minimum 5-7 years of relevant experience.

Desired Experience

Demonstrated experience with the Sponsor’s data environment.
Demonstrated experience exhibiting strong coordination and collaboration skills.
Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets.
Demonstrated experience communicating technical concepts to non-technical audiences.

Equal Opportunity Employer/Veterans/Disabled

Accommodations:

If you are a qualified individual with a disability or a disabled veteran, you may request a reasonable accommodation if you are unable or limited in your ability to access job openings or apply for a job on this site as a result of your disability. You can request reasonable accommodations by contacting Talent Acquisition at Talent-Acquisition@redhorsecorp.com

Redhorse Corporation shall, in its discretion, modify or adjust the position to meet Redhorse’s changing needs.

This job description is not a contract and may be adjusted as deemed appropriate in Redhorse’s sole discretion.