Crypto-Mining Attacks Targeting Kubernetes Clusters via Kubeflow Instances

Cybersecurity researchers on Tuesday disclosed a new large-scale
campaign targeting Kubeflow deployments to run malicious
cryptocurrency mining containers.

The campaign involved deploying TensorFlow^[1]
pods on Kubernetes clusters, with the pods running legitimate
TensorFlow images^[2]
from the official Docker Hub account. However, the container images
were configured to execute rogue commands that mine cryptocurrency.
Microsoft said the deployments witnessed an uptick towards the end
of May.

Kubeflow^[3]
is an open-source machine learning platform designed to deploy
machine learning workflows on Kubernetes^[4], an orchestration
service used for managing and scaling containerized workloads
across a cluster of machines.

The deployment, in itself, was achieved by taking advantage of
Kubeflow, which exposes its UI functionality via a dashboard that
is deployed in the cluster. In the attack observed by Microsoft,
the adversaries used the centralized dashboard as an ingress point
to create a pipeline to run TensorFlow images that perform
cryptocurrency mining tasks.

The intrusions also echo similar attacks^[5]
observed by Microsoft’s Azure Security Center last April that
abused Internet-exposed Kubeflow dashboards to deploy a backdoor
container for a crypto-mining campaign.

“The burst of deployments on the various clusters was
simultaneous. This indicates that the attackers scanned those
clusters in advance and maintained a list of potential targets,
which were later attacked on the same time,” Microsoft’s Senior
Security Research Engineer Yossi Weizman said^[6]
in a report.

The ongoing attacks are said to have used two different
TensorFlow images — tagged “latest” and “latest-gpu” — to run the
malicious code. Utilizing legitimate TensorFlow images is also a
clever design to avoid detection in that TensorFlow containers are
prevalent in machine learning-based workloads.

Furthermore, Microsoft said the attackers are able to take
advantage of the images to run GPU tasks using CUDA, thereby
enabling the adversary to “maximize the mining gains from the
host.”

“As part of the attacking flow, the attackers also deployed [a] reconnaissance container that queries information about the
environment such as GPU and CPU information, as preparation for the
mining activity,” Weizman said. “This also ran from a TensorFlow
container.”

The development comes days after Palo Alto Networks’ Unit 42
threat intelligence team disclosed a brand new form of malware
called Siloscope^[7]
designed to compromise Kubernetes clusters through Windows
containers.

Users running Kubeflow are recommended to ensure that the
centralized dashboard isn’t insecurely exposed to the Internet, and
if deemed necessary, require that they be protected behind
authentication barriers.

Microsoft has also published a threat matrix for Kubernetes^[8] to better understand the
attack surface of containerized environments and assist
organizations in identifying current gaps in their defenses to
secure against threats targeting Kubernetes.

Earlier this April, the company, alongside other members of
Center for Threat-Informed Defense teamed up to release what’s
called the ATT&CK for Containers matrix^[9] that builds upon the
Kubernetes threat matrix to detect “risks associated with
containers, including misconfigurations that are often the initial
vector for attacks, as well as the specific implementation of
attack techniques in the wild.”