As cloud migrations become more widespread and complex, IT security has never been more difficult. 

Enterprise workloads today are rapidly changing to a hybrid mix of on-premise, private cloud and public cloud – not to mention the increasing interest in containers. Getting a handle on it all requires deep visibility, from both the data and device side.

One relatively traditional way of doing this is through log management. The promise, of real-time auditing and tracking across logs generated through all company devices, is impressive, but the reality can be somewhat different. Even before you get to the analysis stage, the sheer volume of data and the speed at which it is produced – if you want to do something, do it properly – can be a headache, especially if it is in different formats.

Morten Gram (left), executive vice president at Humio, notes the issues organisations face, but explains that while the application and infrastructure landscape is changing, so too is log management software. 

“If you want to log everything at scale with a traditional solution, you’re going to have some challenges,” Gram tells CloudTech. “The biggest challenge companies have around this is justifying the cost. You have platforms out there that potentially can scale, but the infrastructure cost, or the storage cost, gets too expensive.”

Alongside infrastructure and storage, Gram says licensing and operational costs are where organisations need to look. But the upsides are there. “If you go back 20 years, the strategy was to log everything – but back then we didn’t have the hardware to cope with it,” he says. “Today, there’s good development within new hardware and infrastructure that actually does make it possible. You can have a huge amount of data in-memory, which makes it possible to work with this huge amount of data at a reasonable cost.”

Having the capability for in-memory data processing doesn’t mean you’re automatically there, however. Database admins will know the sinking feeling of being constrained, of the impact on performance as traffic increases. By purposely building its product for logging, focusing on data streaming, and not bashing people over the head with indexing – ‘the only thing we index is a timestamp’, says Gram – Humio aims for more streamlined performance.

This has a knock-on effect as well. “Being able to ask anything, and getting instant response, is key,” says Gram. “It’s key when it comes to encouraging people to go and investigate and play around and explore the data. 

“If you’re asking the first question, you didn’t know what the second question is until you get an answer back,” he adds. “If you have to sit and wait 15, 20, 30 seconds, it’s not motivating you to ask one more question. But if you get instant responses, it encourages you to ask the second question, and the third, and the fourth.”

Yet certain scalability can only get you so far – which is why Humio is looking to bolster what it calls its ‘unlimited’ offering. Two years ago, the company launched its unlimited plan for self-hosted environments. Later this year, if all goes to plan, the cloud version will be launched. 

Gram explains the rationale. “We have one customer that is running in Google [Cloud], and because of the volumes of data they want to store, and the retention, they need to keep it due to compliance. They were actually in the situation where they needed to buy additional servers just to cope with the storage – not because they needed more compute power.” 

“Working in the cloud, everything in the cloud is dynamic – so you spin up a server, you take it down where it goes down, everything on that disk is gone – so you need to have somewhere else where you have this persistent data.”

An example of the cloud’s dynamism and ever-changing appearance is Kubernetes, or another container platform of choice. Creating environments to build applications more quickly can help companies be more proactive, but it will almost certainly lead to more security holes opening up.

Humio has been a familiar face at KubeCon and has extensive documentation on integration, so Gram knows of what he speaks. “I do think one of the big challenges is the thinking around working with containers, and the cultural shift of going from this more monolithic, static world, to this very dynamic world,” he explains. “And one of the challenges on that journey is actually getting people on the right track.”

“A big challenge is the whole concept of architectural thinking, because you might want to run Kubernetes and then integrate it into an existing environment with old tools that are not built for those dynamics,” Gram adds. “You’re getting people to the right competence level, and then also understanding the impact and consequences of moving into this type of world.”

The promise of transformation if you get everything right therefore remains an alluring one – but as can be seen, larger-scale thinking may have to be employed. Gram notes another customer, Lunar, a Nordic bank, has gone fully digital. “They adopted the cloud-native journey some years ago,” explains Gram. “They are a fintech company; a digital bank. There’s no physical channels at all. If an app doesn’t work, you don’t call into support and ask them to fix it; you go and find another app.”

“For them, it’s really key that they can see when any customer goes on their app, they can, in real-time, monitor what’s going on, so they can be proactive.”

Photo by Harm Weustink on Unsplash

Interested in hearing industry leaders discuss subjects like this and sharing their experiences and use-cases? Attend the Cyber Security & Cloud Expo World Series with upcoming events in Silicon Valley, London and Amsterdam to learn more.