Building a new data ecosystem part 2 - Implementation

In our last few posts, we introduced you to our ecosystem design principles and the platforms we chose. Implementation can always be more perfect, and just as we changed a lot internally in the last 5 months, the next year will hold even more surprises for us. We know you are going through similar changes, so this is  of how we formed an architecture for our data ecosystem.

About us

For those unfamiliar with our team, are around 150 people hailing from all parts of Argentina and the United States. Remote communication is the lifeblood of our company. From chatrooms to dashboards, we're always trying to find the next best way to keep on top of what's going on.

Our data, end to end

Through the evolution, our system architecture can be split into three major parts: Interaction, Data Management, and Analysis. Each area has it's own priority and concerns, but is open to all functions of the business. The only limiting factor is learning some new skills.

Interaction: our digital workplace

Despite being physically apart, we are very much together across our digital platforms. Sometimes, we get stuck using some software that we're used to we can't clearly see that it's no longer a fit for our workflow. Take an elevated perspective and see where these moments could be happening in your flows. 

It may look like we are using a lot of platforms, but that's because we need to do a lot of stuff! Each platform is itself undergoing changes as we rework and make our protocols more efficient. It takes a coordinated team to get all the data lined up the right way, but our team knows it starts with them. As process becomes embedded in systems, manual becomes automatic.

One lens we like to think through is creating digital analogs of physical spaces. What is that digital version of a water cooler or a stickies board? How can we rally our team around these tools to enable our work? As we test different workflows and tools, we match what they do to how we work.

Data Management: make the complex simple

Getting all the data together could be difficult depending on which platforms you chose. Luckily for us, the platforms we chose have pretty good adoption and there's plenty of services like Fivetran that help us bring everything into our data warehouse. Some of the things that we can't do through Fivetran, or need some special care before we upload, we can do so through Google Apps Scripts.

BigQuery is more powerful than our immediate needs, but it's something that accommodates a lot of potential if we decide to scale that way. I'm definitely a beginner at this whole data thing, but the biggest win with BigQuery from a management viewpoint is how easy it is to pass data to downstream users. We iterate fast and create usable tools as we talk about them.

Analysis: from reaction to reflex

In what we've experienced so far, the bulk of the work is in tuning platforms to fluidly flow with people process. Once those connections are well made, the down stream user experience becomes smooth and serene. With our architecture mostly in place, we see the fruits of our labor in how quickly we can act on the latest piece of news.

With each automated report, our sensitivity increases and more time is given to innovation. Reporting perspectives are a representation of our need to understand this organization. Our smoother data flow lets us start knocking out manual processes one by one. We are building towards everything we need being a few less clicks away.

Finding our balance

Our team is growing from users to builders. As they build more around their own workflows, our whole system becomes less tangled. It's been incredible to watch our team pick up new skills and build tools of their own. Beyond just data integrations, our teams are building all kinds of things to make our work better.