Software Engineer - Big Data Ingestion and Processing

1 Month ago • 5-7 Years

Job Summary

Job Description

As a Software Engineer, you will be instrumental in managing large datasets for a critical intelligence mission. Your key responsibilities include loading large datasets, developing ingestion algorithms, optimizing data ingest processes, and creating Apache NiFi schemas. You will develop software tools for preprocessing, modifying, and archiving data in near real-time. Additionally, you'll ensure proper access controls, generate metrics for data integrity, and document data flows. You'll collaborate with data scientists, analysts, and managers in a dynamic environment.
Must have:
  • Experience in Computer Science or related field.
  • Experience with AWS cloud services.
  • Experience working with Databricks.
  • Understanding of SQL database structures.
  • Experience working with Apache NiFi.
  • Experience with large data clusters.
  • Experience with API development techniques.
  • Experience developing ETL processes.
  • Experience creating OS scripts for ETL.
  • Experience with Git version control.
  • Experience testing software solutions.
  • Experience implementing multiprocessing data-flows.

Job Details

About the Organization
Now is a great time to join Redhorse Corporation. Redhorse specializes in developing and implementing creative strategies and solutions with private, state, and federal customers in the areas of cultural and environmental resources services, climate and energy change, information technology, and intelligence services. We are hiring creative, motivated, and talented people with a passion for doing what's right, what's smart, and what works.

About the Role
Redhorse is transforming how government agencies leverage data and technology. We are seeking a highly skilled Software Engineer to join our team supporting a critical intelligence mission. You will play a vital role in ingesting, processing, and analyzing massive datasets, directly impacting the Sponsor's ability to address pressing intelligence questions. You will work with cutting-edge technologies in a dynamic, collaborative environment, directly contributing to national security.

Key Responsibilities

    • Load large datasets into the Sponsor’s on-premises and Cloud environments.
    • Develop and maintain ingestion algorithms and schemas for large datasets.
    • Analyze new large-volume datasets to optimize the data ingest processes.
    • Support the creation of Apache NiFi schemas for new data loads.
    • Develop software tools that efficiently preprocess, modify, aggregate, load, index, and archive large data collections into clusters in near real-time.
    • Ensure proper access controls are implemented.
    • Generate metrics to track data ingest statistics to maintain data integrity and provenance.
    • Document the data-flows according to standards set by the Sponsor.
    • Engage regularly with data scientists, analysts, and managers.

Required Experience/Clearance

    • Demonstrated professional experience in Computer Science, Computer Engineering, Systems Engineering, or closely related discipline.
    • Demonstrated professional experience with AWS cloud services, including long-term storage options, and cloud-based database services.
    • Demonstrated experience working with Databricks.
    • Demonstrated experience understanding SQL database structures and mapping them between different SQL databases.
    • Demonstrated professional experience working with Apache NiFi.
    • Demonstrated professional experience working with large data and high-performance compute clusters such as Hadoop or similar.
    • Demonstrated experience with API development techniques.
    • Demonstrated experience developing and deploying ETL processes for large data sets.
    • Demonstrated experience creating operating system level scripts to perform ETL operations on SQL databases.
    • Demonstrated professional experience with version control systems, preferably Git.
    • Demonstrated experience testing the development of software solutions for the extraction, transformation, and loading of data using the most efficient languages for the task such as NiFi, Python, and SQL.
    • Demonstrated experience implementing multiprocessing data-flows to parallelize ingest operations.
    • Minimum 5-7 years of relevant experience.

Desired Experience

    • Demonstrated experience with the Sponsor’s data environment.
    • Demonstrated experience exhibiting strong coordination and collaboration skills.
    • Demonstrated experience working with full-stack developers to deploy applications that leverage large data sets.
    • Demonstrated experience communicating technical concepts to non-technical audiences.
Equal Opportunity Employer/Veterans/Disabled 
 
Accommodations:
If you are a qualified individual with a disability or a disabled veteran, you may request a reasonable accommodation if you are unable or limited in your ability to access job openings or apply for a job on this site as a result of your disability. You can request reasonable accommodations by contacting Talent Acquisition at Talent-Acquisition@redhorsecorp.com
 
Redhorse Corporation shall, in its discretion, modify or adjust the position to meet Redhorse’s changing needs.
This job description is not a contract and may be adjusted as deemed appropriate in Redhorse’s sole discretion.

Similar Jobs

Vercel - Senior Marketplace Partner Manager

Vercel

San Francisco, California, United States (Hybrid)
1 Month ago
PwC - Azure Data Engineer

PwC

Mumbai, Maharashtra, India (On-Site)
1 Month ago
SSC Technologies - Client Experience Account Manager

SSC Technologies

Melbourne, Victoria, Australia (Hybrid)
1 Month ago
PwC - FY24 - Senior Associate - Risk Assurance - Digital Audit

PwC

Jakarta, Jakarta, Indonesia (On-Site)
9 Months ago
Dentsu - Audience Platform Lead

Dentsu

Bengaluru, Karnataka, India (On-Site)
3 Weeks ago

Get notifed when new similar jobs are uploaded

Similar Skill Jobs

Alpha Sense - Staff Product Designer

Alpha Sense

New York, United States (Remote)
1 Month ago
Lionsgate - Sr. Financial Analyst, Corporate FP&A

Lionsgate

Santa Monica, California, United States (On-Site)
2 Months ago
DevRev - Customer Success Manager

DevRev

Buenos Aires, Buenos Aires, Argentina (On-Site)
1 Month ago
Axon - Senior Project Manager, Commercial Operations

Axon

Scottsdale, Arizona, United States (On-Site)
1 Month ago
Spaulding Ridge - Anaplan Senior Associate

Spaulding Ridge

Bengaluru, Karnataka, India (Hybrid)
2 Months ago
Tencent - Senior Researcher, Natural Language Processing

Tencent

(On-Site)
4 Months ago
Postman - Senior Marketing Data Manager

Postman

San Francisco, California, United States (Hybrid)
2 Months ago
N-ix - Senior DS/AI Engineer

N-ix

Poland (Hybrid)
2 Months ago
Bluevine India - Payment Operations Analyst II

Bluevine India

Bengaluru, Karnataka, India (Hybrid)
1 Month ago
Springer Group - Commissioning Editor

Springer Group

Beijing, China (Hybrid)
3 Weeks ago

Get notifed when new similar jobs are uploaded

Jobs in Herndon, Virginia, United States

Rackspace Technology - Government Services Sales Executive VI

Rackspace Technology

Sacramento, California, United States (Remote)
5 Months ago
Inworld AI - Staff C++ Engineer

Inworld AI

Mountain View, California, United States (On-Site)
3 Months ago
HCL Tech - Network Engineer/L2

HCL Tech

Cary, North Carolina, United States (On-Site)
1 Month ago
Google - Software Engineer III, Infrastructure, Google Cloud Global Networking

Google

Atlanta, Georgia, United States (On-Site)
2 Months ago
Whatnot - Technical Program Manager, Infrastructure

Whatnot

San Francisco, California, United States (On-Site)
2 Months ago
attentive - Account Manager, Mid-Market Sales

attentive

United States (Remote)
1 Month ago
Games For Love - Scholars Program Coordinator

Games For Love

Washington, United States (Remote)
3 Months ago
HCL Tech - Technical specialist - embedded os

HCL Tech

New York, United States (On-Site)
1 Month ago
Blitz app - Director of Product

Blitz app

Los Angeles, California, United States (On-Site)
9 Months ago
Interactive Brokers - Bilingual Client Technical Support

Interactive Brokers

Chicago, Illinois, United States (Hybrid)
1 Month ago

Get notifed when new similar jobs are uploaded

Similar Category Jobs

Looks like we're out of matches

Set up an alert and we'll send you similar jobs the moment they appear!

About The Company

Arlington, Virginia, United States (On-Site)

Arlington, Virginia, United States (On-Site)

Clarksburg, West Virginia, United States (On-Site)

Clarksburg, West Virginia, United States (On-Site)

United States (Remote)

Falls Church, Virginia, United States (On-Site)

Falls Church, Virginia, United States (On-Site)

Arlington, Virginia, United States (On-Site)

Arlington, Virginia, United States (Hybrid)

Fort Meade, Maryland, United States (On-Site)

View All Jobs

Get notified when new jobs are added by Redhorse Corp

Level Up Your Career in Game Development!

Transform Your Passion into Profession with Our Comprehensive Courses for Aspiring Game Developers.

Job Common Plug