Posted on by Douglas Muth

Notes from February 2015 Philly DevOps Meetup: Security Practices for DevOps Teams

As a service to the Philly tech community (and because folks asked), I took notes at tonight’s presentation, called “Security Practices for DevOps Teams”. It was presented by Chris Merrick, VP of Engineering at RJMetrics.

Security is a “cursed role”

  • …in the sense that if you’re doing a really good job as a security engineer, no one knows you exist.
    • It isn’t sexy
    • It’s hard to quantify
    • It’s never done

As DevOps engineers, we are all de facto security engineers

Some tips to avoid ending up like this [Picture of a dismembered C3PO]

Sense. This picture makes none. 


  • Security Principles
    • Obscurity is not Security
      • “A secret endpoint on your website is not security”
      • “Don’t rely on randomness to secure things”
    • Least Privilege
      • Do not give more privileges than are needed
    • Weakest Link
      • If you talk to an insecure system, you’re at risk
    • Inevitability

Security Types

  • Physical
    • Stealing laptops
    • Breaking into datacenters
  • Application
    • Any vector that comes through an application you developed
      • XSS
  • Network*
  • Systems*
    • Applications you didn’t write
  • Human
    • Phishing, social engineering

Server Auth

  • Reminder:
    • Authentication is who you are
    • Authorization is what you can access
  • Don’t access production directory
    • Good news: this is our job anyways
  • Don’t spread private keys around
    • Don’t put in your Dropbox
    • Don’t let it leave the machine you generated it on
    • Use SSH agent forwarding
      • ssh-add
      • ssh -A you@remote
      • ssh-add -l
  • Don’t use shared accounts
    • Especially root
  • Be able to revoke access quickly
    • Time yourself. Go.
  • We use Amazon OpsWorks to help us achieve these goals
    • Chef+AWS, with some neat tricks: simple autoscaling, application deployment, and SSH user management

Logging

  • “Logs are your lifeline”
  • When you get into a high pressure security investigation, you start with your logs
  • Capture all authentication events, privilege, escalations, and state changes.
    • From your Os and all running applications
  • Make sure you can trust your logs
    • Remember – they’re your lifeline
  • Have a retention policy
    • We keep 30 days “hot”, 90 days “cold”
  • Logging – ELK
    • We use ELK for hot log searching
    • Kibana creates logs and lets you monitor your application in real time

Deployment

  • Keep unencrypted secrets out of code
    • Otherwise, a MongoLab exploit becomes your exploit
  • Don’t keep old code around
  • Make deployment and rollback easy
    • More good news: this is our job anyways
    • When dealing with a security issue, the last thing we need a “hard last step” in order to get the fix out
  • IAM
    • Don’t use your root account, ever.
      • Set a long password and lock it away
  • Set a strong password policy and require MFA
  • Don’t create API keys where API access isn’t needed
    • Same goes for a console password
  • Use Managed Policies
    • To make management easier
  • Use Roles to gran taccess to other systems
    • No need to deploy keys, auto-rotates
  • IAM Policy Pro Tips
    • Don’t use explicit DENY policies
      • Keep in mind that everything is denied by default
    • Don’t assume your custom policy is correct just because it saves – the interface only confirms the JSON is valid
    • Use the policy simulator
  • Know Thy Enemy
    • People are out there scanning for AWS keys – treat your private key like a private SSH key

Bonus Tips

  • Set up a security page
  • Sign up for the US-CERT email list, and the security notification list for your OS
  • Other resources
    • OWASP – owasp.org
    • SecLists.org
    • Common Weakness Enumeration – cwe.mitre.org

Conclusion

  • What else should we be doing to keep our work secure? [Picture of C3PO in a field full of flowers]