How Hex Technologies ships fast and scales an effective engineering team

I had the honor of interviewing two of the founders at Hex, Caitlin Colgrove, CTO, and Glen Takahashi, Chief Architect. For those of you who haven’t heard of Hex, you will. Hex Technologies is a platform for collaborative data science and analytics. Hex brings SQL, Python, R, and no-code workflows together in powerful notebooks so that their customers can do exploratory analysis with minimal friction and publish projects as interactive data apps that anyone in an organization can use, from data scientists and data analysts to the C-suite. 

Bridging the gap between a data scientist and a non-technical user is an incredibly difficult problem to solve. The Hex solution pairs code with powerful UI tools to make data more accessible and impactful through a “low-floor, high ceiling” approach. Hex makes data more inclusive and powerful than ever with real-time collaboration, no-code charts, first-class SQL support, and a reactive, graph-based compute engine. On March 22, 2022, they raised their Series B financing—led by Andreessen Horowitz (a16z), Databricks, Snowflake, and existing investors. 

Learn more about the Hex founders and how they use New Relic to debug faster and scale an effective engineering team with New Relic in this New Relic for Startups Q&A. 

Inspiration

Dayna: Where did the inspiration come from to create Hex?

Caitlin Colgrove: Hex came from lived problems that all of us experienced. Barry, Glen, and I all worked together in a previous life on a variety of data analytics products and then went on to observe what I would say is the development of the modern data stack.  In this emerging modern data stack, which is really driven by SQL, Python, and other technical, code-based technologies, we saw a gap in tools that support cloud-driven analytics workflows.

We’ve evolved from finding a solution to share data scientists’ Jupyter notebooks with anyone in different business departments to delivering a collaborative platform for data analytics that covers an end-to-end workflow around exploratory analytics, sharing and consumption, and knowledge management. 

Dayna: Have you always been interested in founding a company?

Caitlin Colgrove: It was never a primary goal of mine to be a founder. I've always wanted to just build great products. I come from a product engineering background and when Barry and Glen came to me with this idea, I couldn't say no. It was such a great team, and it was a space that clearly had a need. I was pretty familiar with it and knew I could have some added value there. It wasn't my goal to start a company. But here we are.

Glen Takahashi: I had definitely thought about it at some point in time, not necessarily because I wanted to start a company, but particularly because I love doing full stack work and I also love building things from scratch. I just find it fun to get to really think through everything from the ground up and figure out what is the right way to build a product. You're not stuck into any paradigms or any existing legacy technologies, which was really exciting. And that's the kind of stuff I love doing.

Product market fit

Dayna: How do you determine product market fit? 

Caitlin Colgrove: It's kind of like the difference between riding a bike slightly uphill versus slightly downhill. You could almost feel the gravity shift at a certain point, and that happened for us early last year. We had key insights into the problems and user demographics, which allowed us to bike downhill rather than uphill against gravity, if you will, in terms of trying to sell the product to people. And at that point, I think it was pretty clear that we had some strong elements of product market fit.

I do think however that there are degrees of fit and we are not done. There's a lot more room to explore in this space so I don't love saying that we have product market fit from that perspective.

Main challenges

Dayna: Obviously it's really hard to build a company, a product from the ground up. As founders, what are some of the main challenges that you face? 

Caitlin Colgrove: As a CTO, I've had a lot of different roles. I was an engineer in the early days of the company. I was running code, but I've moved much more into the business and team-building side of things. That's the challenge I think about the most.

We've established a bit of a reputation for shipping very quickly. New Relic’s fast, intuitive, and customizable UI and flexible pricing play a big role in our ability to ship quickly, monitor releases, and build and scale an effective engineering team with a really strong, positive culture. 

Glen Takahashi: I'd say one of the things that's been the biggest challenge has been building the right documentation and tooling, to have engineers contribute at all parts of the stack. Most engineers are full stack and the fact that they touch everything, from Terraform and AWS infrastructure all the way up to front end code, makes it important to organize and design a team, processes, etc., to ensure all the information is on paper so people can easily make changes anywhere in the stack. All the teams have to work in harmony for this product to properly function.

Caitlin Colgrove: I was going to say yeah, definitely the complexity and the full stack nature of our product, like Glen said, we're basically exposing infrastructure to end users. Obviously, there are controls around that and there are some layers in there, but it makes it a pretty unique development challenge, especially around onboarding engineers.

Using New Relic

Dayna: You mentioned that you have gained this reputation of shipping really quickly and that collaboration is truly needed to help with this complex stack. Can you explain how and why you use New Relic?

Glen Takahashi: We started with New Relic when we began to scale our monitoring infrastructure. Once we started to get past a couple of users and get a lot of customers on board, we needed to get more monitoring to help debug support issues to make sure the stack was stable. We had experience using Datadog in the past, but we weren't particularly fond of it. We kind of just did a search using Google to investigate. It ultimately came down to New Relic and Datadog, and pricing was a very important aspect of this. We have thousands of containers in Kubernetes, and Datadog was going to charge us per container. And New Relic was going to charge us on total data, which was definitely what we wanted and the pricing was thousands of dollars a month difference. 

We ultimately ended up with New Relic and actually, we're pretty happy with it, not only because of pricing but also because of its features. It has really good APM, which we’ve already integrated in our systems and we want to continue to extend the integrations. We also found that its logging and UI are actually better and more intuitive than Datadog.

Dayna: I would love for you to talk about some of the benefits you all saw in the UI for logs and how you decided to implement APM. 

Glen Takahashi: Prior to this, we had just been using CloudWatch to do logging. And CloudWatch, while it has some structured logging, it was not as clean to integrate with and the UI was much more clunky. The thing that we've definitely made the most use of is the structured logging in New Relic. Because of how fast it is, I make daily use of searching for specific transaction types, doing click-arounds to show surrounding logs based on either the pod name, container name or certain operations or trace IDs. We run trace IDs through everything.

We’re also able to customize the views for our custom multiplayer framework for doing real time collaboration using New Relic. It's very easy for us to just add a column, just like the name of the multiplayer operation, whatever. Ultimately the searching is exceptionally fast and debugging things is much better than it was in any of the other systems we've had before.

Caitlin Colgrove: We've actually given all of our sales engineers and support engineers New Relic access. I noticed that New Relic added a user tier recently that I think is aimed exactly at this use case. It's very telling of the sort of accessibility of the UI you get. They mostly use it for logging to see trace IDs, which we're piping through the whole stack out in the field. It's pretty straightforward for them to pop into New Relic and actually take a look at what's going on and be able to identify the relevant stack trace.

It’s easy to turn directly into New Relic to identify whether or not there is a user error or that there needs to be a configuration fix, which gives the engineering team a big head start to identify and resolve problems. This has been really awesome for that side of the company.