SCALE team member Gerald Fontejon met with Chris Nolan (eHarmony) to discuss DevOps and the nVentory project. Chris Nolan and his colleagues will be presenting on nVentory at DevOps Day Los Angeles on Friday January 20th.
SCALE:From looking at who is presenting the topic "nVentory - Your Infrastructure's Source of Truth", we have a Director, System Administrator, and Developer presenting - how does each role play a part with the implementation, management, and daily-use of nVentory?
Chris: I initially implemented nVentory shortly I after I started at eHarmony 18 months ago. There was no easy way to track all of our servers and it was frustrating coming in new and not knowing what was in our datacenters. Ongoing I have become more of an evangelist of nVentory, both internally to eHarmony and externally to the internet community.
Darren, as the engineer, works on features and bugs for nVentory. He is an active developer and sends a majority of his work back to the community, allowing others to share in our new features and tools.
Jeff, as a SysAdmin, is a big consumer and user of the information stored in nVentory.
SCALE: Also could you describe your environment at eHarmony in which you guys use nVentory?
Chris: Our environment is a mix of physical and virtual machines, or nodes, with hundreds of applications and data stores. We use it to manager our thousands of nodes (both physical and virtual) and layer metadata on top of the nodes and nodegroups to manage apps, monitoring, release, etc.
SCALE: It does give you some flexibility with the tool having Darren Dao adding new features to fit your team's needs.
Chris: Definitely. Having the developer of a tool in house is great; especially someone as gifted as Darren.
SCALE: Are the other tools that are comparable with nVentory?
Chris: For tools that do just what nVentory does there are not a whole lot of open source options out there. Puppet and Chef have a "server" concept that allows you to query information about your hosts, but neither track - storage, VIPs, network ports, etc. like nVentory does. There are enterprise solutions but nobody likes shelling out wheelbarrows of cash and working with stuffy consultants. :)
SCALE: Would you be able to use another tool like openLDAP to be the "Source of Truth" within your environment by injecting/adding new fields to accommodate your informational needs?
Chris: LDAP doesn't offer an API or an elegant CLI to extract information. It also doesn't have all the fields or an easy way to layer metadata on top of the nodes it may be tracking.
SCALE: I noticed from poking around LinkedIn, all the members possibly met at ATT Interactive, and had some experience with nVentory in the ATT environment. From your experience as a team of implementing nVentory within eHarmony, what were the major challenges?
Chris: Jeff, Darren and I actually met at a startup that did out of home advertising on digital billboards and TV screens.
Though I wouldn't call it a major challenge, the long haul has been getting people to use it. Sometimes people have to ask a few times about what node is handling service X or what port node Z is plugged into. When you keep telling folks to use nVentory and get the data there, they eventually get it and are more productive in the end. It also results in fewer shoulder taps for the Sysadmins, which is always a good thing.
SCALE: Agreed. Always good to provide users the necessary tools to find information on the given infrastructure. And having the information accessible by everyone outside the SysAdmin team is awesome. Researching the benefits of a "Source of Truth" concept, and nVentory - I'm thinking the number of its implementations will grow. However not many companies have thousands of nodes in their datacenter. What number of nodes would you say is the threshold to make a tech dept. invest some man hours on its 'source of truth' utility implementation? And given your experience with nVentory, what would be some of the crucial steps in order to implement nVentory. The datacenter environment touches multiple teams (netops, sysadmins, security, appops), so I would assume there has to be a lot of coordination.
Chris: I think this question can be asked of all sorts of "devops" tools. When is the best time to implement configuration management? When should I start using Jenkins or a similar Continuous Integration tool? When should I start doing test driven development? And finally, how large should my infrastructure be before I start using nVentory?
If you are only managing 1-10 hosts and don't expect to scale that then you probably don't have any problems that devops (and nVentory) address. If you have 10+ hosts and you expect to scale that as you grow then you should implement nVentory now (as well as the surrounding tools and practices I mention above).
Coordination between groups is important. For example, a lot of our tools make decisions based on rack location. This, unfortunately, is not automatable and requires that DCOps be onboard to manually enter the information. When people see the benefits of using nVentory they are happy to oblige and do what is necessary to make the data useful and valuable to the company.
SCALE: What do you mean by, "a lot of our tools make decisions based on rack location?" I'm unclear on this. And also a couple of the devops tools you mentioned, such as CM and CI are cloud ready - meaning these tools could work within a company's an EC2 or RackSpace environment. Is nVentory capable of supporting a hybrid physical/virtual datacenter(s)?
Chris: Let me clarify that statement. A lot of our tools use rack location in their configuration and deployment decision making.
One example is our custom private cloud. When a multi-node requests (a single service needing multiple services for load balancing and redundancy) the system decides where to place the VMs to ensure they are on different hypervisors in different racks.
nVentory is capable of supporting hybrid environments. The considerations would be around networking and encrypted tunnels between sites and providers.
SCALE: Well, thank you Chris for the interview and a glimpse of eHarmony's utilization of nVentory. I look forward to checking out your discussion. For anyone that is planning on attending your team's presentation - you have homework to prepare them for your guys' talk? Or will the discussion be relatively self-explanatory?
Chris: It's been fun, Gerald. People should think about the foundation tools they use to manage their infrastructure and come ready with questions. Our talk will be self-explanatory with examples of how we integrate our tools and infrastructure management together using nVentory.