Principal Splunk Engineer

This position will be fully remote and can be hired anywhere in the continental U.S.  

We are seeking a highly skilled and experienced Principal Splunk Engineer to join our IT/Security Operations team. The Principal Splunk Engineer will be responsible for the architecture, deployment, maintenance, and tuning of both cloud and on-premises Splunk environments. This role will play a crucial part in security operations, helping to ensure the effectiveness and efficiency of Splunk deployment in monitoring and securing client IT infrastructure. The primary focus for this role is to act as a Subject Matter Expert for Splunk and be able to configure, manage, operate and administrate the platform for managed SIEM. The Principal Splunk Engineer is a member of the Advanced Fusion Center (AFC) team, and will be responsible for following Optiv AFC processes & procedures, as well as managing and maintaining security systems across internal and client environments. The Principal Splunk Engineer will work closely with client Management, Solution Architects, Security Engineers from other internal teams and clients to complete high profile, critical services to existing AFC clients. They will serve as a primary responder for AFC customer systems, taking ownership of client configuration issues, and tracking through resolution.   

Applicants should have a comprehensive understanding of security technologies, including cloud technologies (e.g. Amazon Web Services, Google Cloud Platform, Azure), have extensive experience interacting with customers and have a solid understanding of information security and networking. Providing SIEM management solutions tailored to the needs of clients. Additionally, this position acts as an escalation point for critical and complex client issues, performs configuration and testing of products, assists with the development of work processes, and trains other team members.   

How you'll make an impact 

• Maintain Splunk deployments in both on-prem environments and cloud-based environments across multiple availability zones.  

• Ensure the Splunk deployment continue to run under optimal conditions  

• Evaluate existing Splunk infrastructure, configuration parameters, indexing, search and query performance tuning, security and administration  

• Maintain appropriate infrastructure to maintain performance and data integrity  

• Keep Splunk infrastructure upgraded (as permitted by the applications reliant on them)  

• Manage scalability requirements to meet development and/or business needs  

• Optimize indexes to meet business needs  

• Troubleshoot various issues exhibited by Splunk infrastructure 

• Work closely with architecture, engineering and development and operations teams and jointly work on key deliverables ensuring production scalability and stability  

• Ensure security of Splunk cluster  

• Follow and implement Splunk best practices  

• Ensure appropriate monitoring & alerting of Splunk component health 

What we're hiring for  

• In-Depth knowledge of Splunk architecture and Cloud operations, and a detailed understanding of computer and network security  

• 7+ year experience with Splunk Administration  

• Direct experience maintaining and integrating Splunk within an operational enterprise information system  

• 7+ years of experience with deploying and using Splunk in Commercial Cloud Platform (ex, AWS, Google Cloud, Azure)  

• Proficiency with MS Office and Internet Navigation required.   

• Excellent written and verbal communication skills required.   

• Outstanding time management and organizational skills required.   

• Previous experience in a professional services or SOC environment required.    

• Experience related to security/infrastructure design, IAM, risk analysis and mitigation, disaster/contingency planning, certification/compliance testing, data loss prevention, Network Security Strategies, Technical Documentation, industry standards such as ITIL, COBIT, ISO standards, PCI, SOX, Rollout/Implementation and User Training/Support preferred.    

• Ability to read, analyze and interpret common scientific and technical journals.   

• Ability to respond to common inquiries or complaints from customers, regulatory agencies, or members of the business community.   

• Ability to write speeches and articles for publication that conform to prescribed style and format.   

• Ability to effectively present information to top management, public groups, and/or boards of directors.   

• Ability to sit for extended periods of time.   

• Ability to input data into computer utilizing hands to finger to tab to different fields to input data.     

• Ability to answer telephone and talk and hear other party.