Big Data Developer - Data Pipelines

Job Location(s) US-National
Posted Date 1 month ago(9/16/2021 3:03 PM)
Job ID
# of Openings
Technology Experts - Technical Consultant



At Perficient you’ll deliver mission-critical technology and business solutions to Fortune 500 companies and some of the most recognized brands on the planet. And you’ll do it with cutting-edge technologies, thanks to our close partnerships with the world’s biggest vendors. Our network of offices across North America, as well as locations in India and China, will give you the opportunity to spread your wings, too.


We’re proud to be publicly recognized as a “Top Workplace” year after year. This is due, in no small part, to our entrepreneurial attitude and collaborative spirit that sets us apart and keeps our colleagues impassioned, driven, and fulfilled.


Perficient currently has a career opportunity for a Big Data Developer We are looking for someone to be based in the US, but are flexible on specific location.  Candidate will be expected to work US East Coast hours with occasional flexibility needed to work with client team based in India.



Job Overview:


One of our large clients is expanding their current data footprint on the cloud to provide analytics, BI and data APIs. Majority of data will be batch processed with data validation, data quality and transformation into a multitude of data platforms such as Redshift, Postgres and Hive.


A Senior Technical Consultant is expected to be knowledgeable in two or more technologies within (a given Solutions/Practice area). The  Senior Technical Consultant is expected to have strong development and programming skills in Spark with a focus on Scala/Java and other ETL development experience in the big data space.  You are expected to be experienced and fluent in agile development and agile tools as well as code repositories and agile SLDC/DevOps frameworks.


You will work with architects and infrastructure teams to develop, test, deploy and troubleshoot your code as well as provide input into solutions and design of the system. You will collaborate with some of the best talent in the industry to create and implement innovative high quality solutions focused on our clients' business needs.


  • Work with data engineering team to define and develop data ingestion, validation, transformation and data engineering code.
  • Develop open source platform components using Hadoop, Spark, Scala, Java, Oozie, Hive and other components
  • Document code artifacts and participate in developing user documentation and run books
  • Troubleshoot deployment to various environments and provide test support.
  • Participate in design sessions, demos and prototype sessions, testing and training workshops with business users and other IT associates


  • At least 3+ years of experience in developing large scale data processing/data storage/data distribution systems, peferably Cloudera
  • At least 3+ years of experience on working with large Hadoop projects using Spark and Python and working with Spark DataFrame, Dataset APIs with SparkSQL as well as RDDs and Scala function literals and closures.
  • Hands-on experience with Hadoop, Hive, Sqoop, Oozie, HDFS.  Great SQL Skills. 
  • Experience with ELT/ETL development, patterns and tooling is recommended
  • Experience with AWS and cloud environments including S3 object storage, EC2, RDS and Redshift is a plus
  • Experience with SQL including Postgres, MySQL RDBMS platforms
  • Experience with Linux (RHEL or Centos preferred) environments.
  • Experience with various IDE and code repositories as well as unit testing frameworks.
  • Experience with code build tools such as Maven.
  • Fundamental knowledge of distributed data processing systems and storage mechanisms.
  • Ability to produce high quality work products under pressure and within deadlines with specific references
  • Strong communication and collaborative skills
  • At least 5+ years of working with large multi-vendor environment with multiple teams and people as a part of the project
  • At least 5+ years of working with a complex Big Data environment
  • 5+ years of experience with JIRA/GitHub/Git and other code management toolsets

Preferred Skills and Education:

Bachelors’s degree in Computer Science or related field

Certification in Spark, AWS or other cloud platform


Perficient full-time employees receive complete and competitive benefits. We offer a collaborative work environment, competitive compensation, generous work/life opportunities and an outstanding benefits package that includes paid time off plus holidays. In addition, all colleagues are eligible for a number of rewards and recognition programs including billable bonus opportunities. Encouraging a healthy work/life balance and providing our colleagues great benefits are just part of what makes Perficient a great place to work. 


More About Perficient


Perficient is the leading digital transformation consulting firm serving Global 2000 and enterprise customers throughout North America. With unparalleled information technology, management consulting and creative capabilities, Perficient and its Perficient Digital agency deliver vision, execution and value with outstanding digital experience, business optimization and industry solutions.


Our work enables clients to improve productivity and competitiveness; grow and strengthen relationships with customers, suppliers and partners; and reduce costs.  Perficient's professionals serve clients from a network of offices across North America and offshore locations in India and China. Traded on the Nasdaq Global Select Market, Perficient is a member of the Russell 2000 index and the S&P SmallCap 600 index.


Perficient is an equal opportunity employer and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national, origin, disability status, protected veteran status, or any other characteristic protected by law.


Disclaimer:  The above statements are not intended to be a complete statement of job content, rather to act as a guide to the essential functions performed by the employee assigned to this classification.  Management retains the discretion to add or change the duties of the position at any time. 




Sorry the Share function is not working properly at this moment. Please refresh the page and try again later.
Share on your newsfeed

Need help finding the right job?

We can recommend jobs specifically for you! Click here to get started.