Job responsibilities of a hadoop admin
With increased adoption of Hadoop in traditional enterprise IT solutions and increased number of Hadoop implementations in production environment, the need for Hadoop Operations and Administration experts to take care of the large Hadoop Clusters is becoming vital.
Job Responsibilities of a Hadoop Administrator: Responsible for implementation and ongoing administration of Hadoop infrastructure. Aligning with the systems engineering team to propose and deploy new hardware and software environments required for Hadoop and to expand existing environments. Working with data delivery teams to setup new Hadoop users. This job includes setting up Linux users, setting up Kerberos principals and testing HDFS, Hive, Pig and MapReduce access for the new users.
Cluster maintenance as well as creation and removal of nodes using tools like Ganglia,Nagios,Cloudera Manager Enterprise, Dell Open Manage and other tools. Performance tuning of Hadoop clusters and Hadoop MapReduce routines. Screen Hadoop cluster job performances and capacity planning Monitor Hadoop cluster connectivity and security Manage and review Hadoop log files. File system management and monitoring. HDFS support and maintenance. Diligently teaming with the infrastructure, network, database, application and business intelligence teams to guaranteehigh data quality and availability.
Collaborating with application teams to install operating system and Hadoop updates, patches, version upgrades when required. Point of Contact for Vendor escalation DBA Responsibilities Performed by Hadoop Administrator: Data modelling, design & implementation based on recognized standards. Software installation and configuration. Database backup and recovery. Database connectivity and security. Performance monitoring and tuning. Disk space management. Software patches and upgrades. Automate manual tasks. DWH Development Responsibilities Performed by Hadoop Administrator: DWH admins job responsibilities includes developing, testing and monitoring batch jobs for the following tasks:
Ensure Referential integrity. Perform primary key execution. Accomplish data restatements. Load large data volumes in a timely manner. Now that you know about the job responsibilities of a Hadoop administrator,let’s take a look at the skills required to be one. Skills Required to become a Hadoop Administrator: General operational expertise such as good troubleshooting skills, understanding of system’s capacity, bottlenecks, basics of memory, CPU, OS, storage, and networks. Hadoop skillslike HBase, Hive, Pig, Mahout, etc. The most essential requirements are: They should be able to deploy Hadoop cluster, add and remove nodes, keep track of jobs, monitor critical parts of the cluster, configure name-node high availability, schedule and configure it and take backups.
Good knowledge of Linux as Hadoop runs on
Linux. Familiarity with open source configuration management and
deployment tools such as Puppet or Chef and Linux scripting.
Knowledge of Troubleshooting Core Java Applications is a
plus.
What exactly is Computer Science ?
Computer Science is the science of using computers to solve problems. Mostly, this involves designing software (computer programs) and addressing fundamental scientific questions about thenature of computation but also involves many aspects of hardware and architecting the large computer systems that form the infrastructure of commercial and government enterprises. Computer scientists work in many different ways: pen-and-paper theoretical work on the foundations and fundamentals, programming work at the computer and collaborative teamwork in doing research and solving problems.
What Computer Science is not
...
Computer Science is not
about using software, such as
spreadsheets (like Excel),word processors (like Word) or image
tools (like Photoshop). Many software packages are complicated to
master (such as Photoshop or Excel) and it is true that many jobs
depend on expertise in using such
tools, but computer science is not about using the tools. It is not
about expertise in computer games, it is not about about writing
content in websites, and it is not about not about assembling
computers or knowing which computers are best buys. Edsger
Dijkstra, a famous award-winning computer scientist once said,
"Computer Science is no more about computers than Astronomy is
about telescopes". Computer Science is about the principles behind
building the above software packages, about the algorithms used in
computer games, about the technology behind the internet and about
the architecture of computing devices.
What is Information Technology, and how
is it different from Computer Science?
While computer science has
become a somewhat precise term as a field of study (like
geology), information technology
(IT) is a somewhat more vague term. The commercial
world uses the term IT in a variety of contexts, generally, to mean
"anything to do with computers". Many business uses of this term
refer specifically to the combination of databases, information
processing systems and communication systems (email, web browsing)
they have been installing in the 80's and 90's. Thus, an IT job
could mean a sales job in a computer company, or a business manager
overseeing the installation of software, or it could mean a network
technician who installs fiber-optic cable, or of course a software
engineer. However, computer science generally denotes a
professional with computer science training, one who is involved in
the creation of software and software systems. Most educational
programs are in computer science, which has a long tradition of
accredition, standardized testing (such as the GRE subject test in
computer science), prestigious research journals and well-defined
curricula. In contrast, while some schools offer IT curricula,
these are less well-defined, and probably not as rigorous as
computer science curricula and degrees.
What is
software?
Computer science is not about building keyboards or monitors
or the cables that connect your PC to your printer. While these are
important to the functioning of a computer, as is electricity,
computer software consists of interacting programs each of which is
a collection
of instructions capable of being
executed on a computer. So, first we need to think of a computer as
a "dumb" machine that knows how to execute elementary
"instructions" (add this, multiply that). Then, software programs
are collections of instructions that achieve higher-level end
objectives. In a sense, the "intelligence" lies in the software and
it is the difficulty of creating reliable, intelligent software
that has made the young discipline of computer science into the
large, diverse field it is today. Software systems now pervade
almost all aspects of life, including high-end entertainment (such
as the computer-generated dinosaurs in Jurassic Park),
mission-critical control systems (factories, robots, aircrafts,
space-travel), information systems (banks, websites, medical
databases, government systems) and research tools (earthquake
simulators, drug-design software, astronomy databases).
What is
programming?
Programming is the intellectual endeavor of creating
individual software programs. Part of it involves thinking (design,
analysis), part of it involves coding (translating a design into
instructions via a programming languages such as Java or C++) and
part of it involves testing (subjecting software to a battery of
tests to make sure it works). Programming has been likened to
mathematics (analytic thinking) to writing (artfully telling a
story), to engineering (building larger software out of smaller
software units) and to art (exercising creativity). The part of
programming that is most easily identified in Hollywood depictions
is coding, the process of typing instructions in a programming
language (such as Java or C++); this involves the stereotypical
hunching over a monitor, pounding at the keyboard and watching the
software execute.
Is Computer Science mostly
programming?
Far from it. Initially, it may seem that it is all about
programming because it is the skill whose teaching we start with
(because it's fun, it's challenging and it's a prerequisite to
further computer science). However, most undergraduate curricula
devote 3 to 4 courses exclusively to programming, leaving 10-15
other computer science courses. Some of these use a student's
programming skills acquired earlier, but most concentrate on some
aspect of computer science central to the discipline. So, what are
these areas of computer science? You can: learn about how computers
are built (architecture), the principles behind important
"infrastructure" software systems (operating systems, databases),
study classic algorithms and learn to design your own, learn how
compilers and language translation is done, study specialized
computer science areas such as artificial intelligence, parallel
computing, networks, graphics, bioinformatics, robotics, education
or multimedia.
What's interesting about Computer
Science?
Why do people find computer science interesting? Initially,
interest usually begins with programming and mastering the many
details and thought processes involved in programming. Later, once
programming is "been there, done that", people get interested in
designing large software systems, or in computer architecture, or
in one of the many specialized areas of computer science. One of
the best aspects of a young discipline is that there are many open
problems awaiting the next generation of computer scientists. For
example, one of the "holy grail" problems in computer science (the
P=NP question) is still unsolved. Many people believe that the
golden age of computing has just begun.
What kinds of careers are open to me with
a degree in Computer Science?
Many people incorrectly believe that a computer science
career is all about programming. While it is true that most
entry-level jobs after a Bachelor's degree involve programming,
most practioners eventually graduate to other responsibilities such
as design, coordination, testing, planning and management. Thus,
you typically start with a software engineering job after a
Bachelor's and move on (after about 5 years of experience) into
higher-level positions. With advanced coursework and a Master's
degree, you can work in an area of specialization that uses your
advanced coursework. For example, working for an animation outfit
such as Disney will require at least 2 to 3 courses in computer
graphics. Finally, a PhD degree usually finds its recipient in a
research environment such as a research lab, research wing of a
large corporation or a university.
Are there careers in Computer Science
that involve people-skills, or will I be staring at a screen all
day?
Most career paths in computer science involve people skills
and interacting with people. Beyond an entry-level position as a
software engineer, almost any corporate position requires working
with people. The creation of software is most often a team effort,
and software companies are organizations of people like any other
type of company. Thus, if your career path is typical, you will not
be alone in your cubicle staring at the screen.
What if I like programming all
day?
There is of course a rich tradition of computer scientists
who love developing software and who are happy spending most of
their time in programming or designing software. Some are so
motivated that they often spend hours on programming beyond their
time at work. Many of these efforts have resulted in the vast
amount of free open-source software available on Linux and other
systems.
What are hot topics in Computer Science
today?
The core areas of computer science,
including software engineering, graphics, networks,
databases,
multimedia and artificial
intelligence remain strong today. At the same time,
some of the most exciting new work in computer science is occurring
at the intersection between computer science and other fields. For
example, computer science is changing the way biological research
is conducted in fundamental ways, leading to a new field
called bioinformatics at
the intersection of biology and computer science. Similarly,
computer simulations are making it possible to study problems in
physics, chemistry, economics and geology that were difficult
without computers.
What are some challenges left open in
Computer Science for my generation?
Some people wonder if all the "important" problems in
computer science have been solved, leaving only tinkering for
future generations. Nothing could be further from the truth.
Perhaps the most important theoretical objective in computer
science (the P=NP question) remains unsolved to this day. Another
seemingly mundane problem shows no sign of being solved: how to
rapidly and easily create large software systems without errors.
Similarly, applications of computer science to other disciplines
have only begun to scratch the surface. Are you interested in these
challenges?
Can Computer Science be combined with
other fields of study?
Computer science is unlike many other disciplines in that
there is a large intersection with other fields. This makes it
possible to "dual-major" in computer science and another field in a
meaningful way. For example:
Is studying Computer Science useful if I
really want to major in something else?
For those disciplines that intersect with computer science
(see above), the answer is clear. But what about History or
English? An honest answer to this question would have to include a
comment from a recent history major: "I really want to study
history, but see computer science as a back up in the job market".
Fair enough. However, consider that computer science also provides
a certain kind of intellectual training, one that is focused on
logical thinking, mastery of detail, teamwork and multiple facets
of problem-solving. Computer science students, for example, find
the logical-thinking tests on the LSAT (Law school exam) and GMAT
(Business school exam) quite easy. Many students bound for med
school also study computer science. And, yes, it helps to have a
backup!
Why is computer science so
hard?
Initially, it does seem that way. The reason is that,
programming is challenging and is introduced "cold" to students in
a first computer science course. Compare this to the study of
mathematics: math is with us since Grade 1 and introduced in small
steps right through school. Programming is a similar intellectual
skill that takes time to master, usually about 4-5 courses. While
there are always students to appear to find programming easy and
unnecessarily intimidate others into believing they are not suited
to computer science, most of us learn skills step-by-step over
time. Can anyone with no musical background learn a musical
instrument in one semester? Can you learn to speak a foreign
language fluently with a single course? Many students tend to give
up because of a combination of "others seem to get it and I don't"
and "why isn't it coming to me?". Any skill acquisition is hard if
viewed negatively. But like any skill acquisition, it can be
acquired with patience and persistence. And once the skills are
acquired, the supposedly "super-smart" kids who "got it" earlier
don't seem that unreachably smart anymore.
What does it take to be successful in
Computer Science?
Computer science is about a unique kind of problem-solving:
creatively solving problems using computation. If you are creative,
if you like puzzles, if you like problem-solving in other domains
(engineering, mathematics, sciences), if you are comfortable with
abstract thinking, if you like working at the intersection of
multiple disciplines - if any of these apply to you, then Computer
Science is for you.
What do I do next if I want a career in
Computer Science?
In high-school, consider taking a few computer science
courses. One advantage of taking the AP exams is that many schools
will transfer these if you have a sufficiently high score. It's
important to note this fact: you do NOT have to have any computer
science in high-school to take it up in college. Universities not
only offer courses for the completely uninitiated but also find
that those with high-school computer science do not necessarily
fare better. Suggested next steps:
CS/IT careers options
There is a common belief that job opportunities for B.Tech Computer Science graduates are restricted to Software Programming. But the fact is that they have a wider job horizon beyond just programming.
Opportunities other than Programming after B.Tech in Computer Science
Bachelor of Technology in Computer Science is a 4 year professional course which helps the aspirants to build up a career in the area of information technology. Apart from Software Programming, there are various job opportunities for these graduates in various levels of application. The most commonly chosen job areas other than Software Programming include:
Systems engineering
Hardware design
Software testing
Electronic data processing
Computer servicing
Database administration
Instructors
Lectures/Professors
Customer support engineers
Different Roles, Different Names
• Software Developers: Software developers are professionals who are concerned with facets of the software development process which involves activities such as design and coding, computer programming, project management, etc.
• Hardware Engineers: These professionals do research, design, develop, test, and oversee the installation of computer hardware which inter alia includes computer chips, circuit boards, systems, modems, keyboards, and printers.
• System Designer: Professionals involved in system designing, Logical & Physical Designing wherein logical designing can be enumerated as the structure & characteristics such as output, input, files, database & procedures, etc.
• System Analyst: Computer engineers who work as systems analyst do research about the existing problems and plan solutions for the problem. They also recommend software and system related problems and coordinate development between business development teams.
• Networking Engineers: Networking engineers are computer professionals involved in designing, implementation, and troubleshooting of computer networks.
• DBA: DBA or Database Administrator are the professionals who are bestowed with the job to design, implement, maintain, and repair an organization's database. Inter alia DBA professionals are also known as Database Coordinator or Database Programmer in IT sector.
Other useful courses to pursue other job profiles
For most of the job profiles the firms demand several certifications as well along with the degree. The networking certifications include:
CCNP
CCNA
CCIE
CCSP
MCSE
CCDA is the most popular certification in designing side. There are various certifications for manual testing and automated testing. QTP certification of HP is the most demanded certification in automated testing.
Apart from this, those who wish to pursue job opportunities in database and ERP can have courses in SAP, Oracle DBA etc. If you want to switch to academics, you should ideally take a post graduation and then appear for UGC exam.
After doing B.Tech in Computer Science you can also look at hardware sector and can have specialization in different hardware components like servers, PC’s etc and can join in as Customer support engineers for various hardware manufacturing firms.
Salary ranges associated with various job profiles
For aspirants in Software Programming, a remuneration of around ₹20000/- per month can be expected in the initial stage. For professionals with certifications in designing and testing a remuneration of around ₹30000/- per month can be expected in the initial stage itself. With sufficient exposure in the field these professionals can expect very good salary packages.
After B.tech from CS & IT
A. Public Sector:
· There are several government jobs available for you in renowned firms like ONGC, DRDO, ISRO, BSNL JTO, BHEL, NTPC, SAIL, CDAC, CDIT, IOCL, IES, IAS, IPS and CDS.
· Most of these companies generally consider the GATE score.
· UPSC and SSC conduct various tests. Tests conducted by SSC are for CBI and are called CGPE exams.
· Public sector banks are also one of the best jobs requiring good mathematical and economical skills. They conduct various tests as well.
· For more information, you can go towww.governmentjobsalerts.com
B. Private Sector:
· IT companies like TCS, Infosys, Wipro, CTS, HCL, Satyam, HP and many more recruit computer science graduates on the basis for their technical and presentation skills, requirement of good GPA is also a necessity but it is more important to achieve the former two skills.
· Besides this, companies such as Tech Support, BPO and call centers offer good salary, as well, and number of perks.
C. Further studies:
· If you wish to study further, then myriad options are available. You can go for MBA, M.tech, MCA and many others. If you wish to study abroad, there is a plenty of information available on www.studyplaces.com and there is an additional requirement to clear IELTS, TOEFL and GRE. For MBA, entrance exams such as CAT and GMAT are the most renowned. Besides, there are others too like MAT, XAT, JMET, FMS, IBST, SNAP and CMAT.
A. NCSA (National Computer Science Academy)
· Through this, you can test your skills with one of their online certification exams.
· You can earn money by helping people get the computer training and certification they need.
· It conducts exams based on- Microsoft Windows, PC Basics, computer hardware, Microsoft Office, Visual C++, HTML and JavaScript.
· Employee testing program allows you to test in any one of their available certifications or to create your own custom exams using their testing engine.
· On Demand Training: It features over 7500 quality instructors who led videos and courseware available online 24/7. Courses also offered individually and on CD.
· You can go to www.ncsacademy.com
B. Microsoft Certification Exams:
· When you pass you first Microsoft Certification Exams, you become a member of the Microsoft Certified Professional (MCP).
· There are wide range of Microsoft Certification Exams for entry level to experienced professionals and for domains ranging from networking, system administration, and database management to programming, web development and other advanced technologies.
It conducts exams like MTA, MCTS, MCIP, MCIP SQL Server, MCPD and for more, you can go to www.microsoft.com/learning/en/us/certification-exams.aspx
· There is a lot more information available on www.cheat-test.com regarding exams.
C. ACM-ICPC(International Collegiate Programming Contest):
· It is a team based annual multi-tiered competitive programming competition among the universities of the world sponsored by IBM.
· You should have good skills of algorithms and programming languages like C++ and Java.
· Find out more on www.icpc.baylor.edu
D. Internship:
· Through internship, you can increase you marketability in the job force.
· There are a few paid internships as well.
· If you are planning to do internship abroad then it brings you plenty of advantages but there are many things to be taken care of like housing safety and visa requirement. Moreover, international resumes, travel insurance (some companies provide this, others don’t), on-site support and cost of living must be considered.
· There are some companies that help you find an international engineering internship:
o DAAD Research Internships in Science and Engineering (RISE)www.daad.de/rise-programs/en/index.html
CDS International Internships Abroad www.cdsintl.org
Global Experiences Engineering Internshipswww.globalexperiences.com/internships/barcelona-spain-internships.php
IEM Internships Abroad www.iemstudyabroad/students/study-abroad-internships/
** Technical Skills: You must have the complete knowledge of the programming language that you have practiced.
** Presentation Skills: All your skills will be wasted, if you don’t know how to present them in front of others. Be confident in what you speak and you need to have the perfect body language.
** Economics: When you have a working knowledge of economics, it’s like having a mild super power.
** CS Fundamentals: Although look quite easy and basic but it is one of the most important factors you’ve to remember. You must have a great practice in data structures and algorithms of the programming language you know.
** HCI (Human Computer Interaction): It is an interdisciplinary field focused on the interactions between human users and computer systems, including the user interface and the underlying processes which produce the interactions. The contributing disciplines include computer science, cognitive science, human factors, software engineering, management science, psychology, sociology and anthropology.
** UCD (User Centred Design): It is an approach to design that grounds the process in information about the people who will use the product. It focuses on users through the planning, design and development of a product.
Besides all these skills, time management, positive attitude and good communication skills are the primary requirements.
Hardware Engineering
Job Description
As this job title implies, a computer hardware engineer's expertise lies with the physical parts of computers and computer systems. Often referred to as a hardware engineer, he or she oversees the manufacture and installation of computer systems, servers, chips and circuit boards as well as peripherals including keyboards, routers and printers. A computer hardware engineer's job includes design, development and testing of computer hardware.
Employment Facts
There were 83,000 computer hardware engineers employed in 2012. One-fifth of people in this occupation work in the computer systems design industry. A slightly smaller number work in the semiconductor and other electronic component manufacturing industry. A significant number work in the computer and peripheral equipment manufacturing and the physical, engineering and life sciences research and development industries.
Educational Requirements
One typically needs a bachelor's degree in computer engineering for this career, but a degree in electrical engineering orcomputer science will also do. If you decide to get an engineering degree, look for a program that isaccredited byABET, a professional engineering association. To find one, you can use the accredited programsearch tool on ABET's website. Since computer hardware engineers need a background in computer science, students who major in electrical engineering should take classes in that subject.
Why Do You Need to Know About Educational Requirements?
Other Requirements
To be successful in this field, one needs certainsoft skills, or personal qualities. A computer engineer must be inquisitive, analytical and detail oriented. Good communication skills, both in written and oral, are also important.
Advancement Opportunities
As a computer hardware engineer gains experience, he or she will begin to work more independently and may eventually begin to supervise other workers. Some becomecomputer and information systems managers.
Why Do You Need to Know About Advancement?
Job Outlook
The US Bureau of Labor Statistics expects employment of computer hardware engineers to grow more slowly than the average for all occupations through 2022. Graduating from an accredited program, as addressed earlier, will increase your chances of finding a job in this tight market as will having a master's degree and experience with software.
Why Do You Need to Know About Job Outlook?
Earnings
Computer hardware engineers earned amedian annual salary of $100,920 in 2012 (US).
Use theSalary Wizard at Salary.com to find out how much a computer hardware engineer currently earns in your city.
A Day in a Computer Hardware Engineer's Life:
These are some typical job duties taken from online ads for computer hardware engineer positions found onIndeed.com:
Diagnose and troubleshoot failed rotational media and solid state storage devices.
Analyze user needs and recommend appropriate hardware.
Modify hardware to either enable forensic analysis of the media or to change the functionality of the hardware for other purposes.
Participate in the design and development of electronic circuits and assemblies.
Conduct technical exploitation and examination of high priority digital media.
Test and peer review for technical accuracy, standards and compliance to the design.
© The Education360 Team