At Atlassian we believe strongly in empowering our developers to own the services they write and run. Ever Atlassian has the permission, and the responsibility, to push their service to production. To do this we make heavy use of Amazon’s IAM service, which has been glued into our corporate LDAP directory.
A core tenet of this pattern is individuals don’t have direct access to root passwords or other critical credentials. Rather, permissions to various resources are assigned to IAM roles and, depending on LDAP group membership you have access to all the resources you need from day one by virtue of joining a team in LDAP. No days wasted waiting for overworked admins to action a ticket to add your ssh key to a jump box.
Atlassian developers are well accustomed to the notion of acquiring a set of permissions by assuming an IAM role for a set period of time. We wanted to continue to provide this experience as we rolled out Kubernetes to Atlassian.
Kubernetes authentication models
Kubernetes provides a rich set of authentication modes including pre shared tokens, static password files, and X.509 certificates.
Pre shared tokens are problematic in our environment as issuing them would require manual intervention from an operator and revoking them currently requires restarting the API server. Neither of these are things we wanted in our environment. The same is true for static password authentication, with the added complexity that we would have to acquire the clear text password of the developer beforehand (which is strongly discouraged in our environment), and has the same revocation drawbacks as the previous method.
As we’re operating in the public cloud, encrypting all the traffic between client, kubelets, etcd, and api servers, is prudent. This meant for each kubernetes cluster we stand up, we have a certificate chain in place to secure and validate the communications between the kubernetes components. This proved to be the key to implementing our authentication mechanism using X.509 certificates.
Introducing kubetoken
Introducing kubetoken, Atlassian’s solution for issuing time-limited SSL certificates for use with Kubernetes clusters.
kubetoken consists of two components; a signing daemon and a command line client. Both components use Active Directory flavoured LDAP to validate group membership.
Gaining access to a Kubernetes cluster proceeds thusly:
- The client presents the user with a list of available roles from active directory.
- Once one is chosen, the client generates a certificate signing request using the client’s username as the CN and the role as the O. (Configuration of the matching RBAC ClusterRoleBinding is outside the scope of this article).
- This request is sent to the signing daemon which validates the requested group’s membership.
- If the request is valid, then the CSR is signed and returned to the client along with metadata like the addresses of the api servers that this certificate is valid for, and for completeness, a copy of the CA public key. This metadata allows kubetoken to configure kubectl to use this set of credentials.
We’ve chosen, somewhat arbitrarily, to issue certificates with a 6 hour validity, after that time they will no longer be accepted by the api server. This automatic expiry was a key reason for choosing X.509 certificates.
Using X.509 certificates proved to be the ideal solution for us as it avoided the requirement, and latency, of distributing tokens to all apiservers. Our apiservers run in an autoscaling group behind an ELB, so may appear without notice complicating promulgation of a set of valid tokens. The user experience of a request sporadically failing because their token hadn’t been synced to the apiserver the request happend to land on would be less than stellar.
X.509 certificates also gave us a reliable expiry mechanism that does not require restarting the apiserver. Expiring tokens by co-ordinating their removal from all apiservers would be complicated, and the risk of this removal process failing thereby leaving credentials valid beyond their expiry window was a strong concern. Using time bounded certificates avoids these expiry headaches.
Sound’s great, where can I get it?
If you’d like to try kubetoken, the source is available on GitHub. Installation instructions are available in the README.