The idea is to have a way for an admins to SSH into a server without having to use passwords. This adds a level of security to your server setup. Without private keys you have to enter your user name and password. This can be less secure than generating SSH keys and adding your public key to SSH, plus with keys you don’t have to remember passwords.
The code to create an ed25519 ssh key in the current users .ssh directory will look like this.
ssh-keygen -f ~/.ssh/key-name -t ed25519
The -f flag tells ssh-keygen the name of the files you want to create. The above command would create key-name(private key) and key-name.pub(public) key, in the current users .ssh directory. The ~ is a Linux shortcut meaning /home/current_user/ so you don’t have to type all that.
The -t flag tells ssh-keygen what type of algorithm to use. If you don’t specify the -f flag and give the file a name, then both files are output in the current users .ssh directory as ed25519 and ed25519.pub
One issue with creating SSH keys is there are so many ways to do it and no one tells you why they do what they do. A quick search will reveal almost everyone has their own way of doing it. If you are new to ssh keys I suggest you read this article really quick.
In the github docs they tell you how to create the ssh keys like this
ed25519 is basically the newest type of key, it is supposed to be the most secure
-C is for adding a comment to the key. This helps you identify it later in places like ~/.ssh/known_hosts, ~/.ssh/authorized_keys and when you use the command ssh-add -L which prints out your public key info
-t specifies the type of key. The above command tells ssh-keygen to create an ed25519 type of key more info
-f /home/akashicseer/tests/ssh/file_name This tells ssh-keygen where to put the file. If you don’t specify the name then it will use a default of something like id_ed25519 for private key and id_ed25519.pub for the public key. The code above will put the files named file_name (private key) and file_name.pub ( public key) in the folder /home/akashicseer/tests/ssh/ If you don’t specify the full path to the exact folder your keys will be put into your users home directory in the default .ssh location. On Ubuntu this is /home/username/.ssh/
NOTE: for ssh deploy keys, don’t specify a passphrase when you create them or you will have to manually enter it later when Packer or whatever you use runs your provisioning code. That means you won’t be able to automate if you enter a passphrase, because it will ask the terminal user to enter the phrase to do a git clone.
There are different types of ssh keys. If you don’t add the ed25519 part then a regular ssh key of type rsa is created, this is the default type of ssh key. Basically Github documentation is showing how to create a secure type of key to use with your code deployments. You will use this key to clone your repository to your server instances.
Creating the key is only half the battle. You must decide how you will create the key, especially if you are automating deployments. When automating deployments the process becomes very complicated.
First either you create the keys you need in the instance you are creating then use the github api to add them to the proper repo. Or you create the keys on a local development computer and use something like Hashicorp Packer to upload the files to the server instance during creation. The latter is the easiest way especially in automation of the infrastructure.
If you are creating your keys locally and using Packer to upload them, you will need to login to Github and go to the deploy keys section of the specific repo to add your public key. The public key is the one that ends in .pub usually. The easiest way to copy the key value is to use xclip which I mention in this article.
If fully automating the process and creating the key on the actual instance, you must remember to eventually remove older keys. Github lets you have like 50 keys per repo max. If your repo needs to be deployed to many instances, such as a microservice structure you can contact them to get added key abilities. You could also reuse the same key, but that would require keeping the private key in a repo as well which probably isn’t a very good idea at all, since ssh keys are the same as passwords basically.
Also remember this. If you are using deploy keys only to deploy by cloning the repo, then deleting the key after the clone is perfectly fine. You only need to use this key one time to clone (aka deploy ) your code, after this it is useless. You can and probably should create new SSH keys for deployment each time and remove them from Github after you deploy, then delete them from the server instance.
Unless you plan on keeping the same instance up and trying to pull from the repo etc. That is messy. Personally I’d prefer to use Packer to create new instances when I need to and redeploy. This has the added benefit that I can upgrade the instance with security etc., test it, add my app code, test it, then swap over after migrating the database and other files. This is like creating a clean slate every time.
You will also need to know how to add the keys to ssh-agent and use them, which I cover in this article.
SSL is a very important subject. All websites/apps should be using it. However the docs will leave you scratching your head saying WTF? So I am creating this long list of resources for anyone else who ever has to learn how to use it.
First here is a link to the docs – this will cause confusion as nothing tells you how to use the pieces together. So it is like looking into a box of legos and knowing it builds something but you don’t even have a picture as a hint. The best you can do is use the pieces to build something that doesn’t even resemble the original creation.
Recently I decided to start automating my infrastructure. Before this it had never occurred to me how stuck in the past our ancient tools are.
These days we have the cloud. We can fire instances up in seconds. But to do this we need ways of automating things. Tools such as SSH, SSL, GIT etc. feel stuck in the 1990’s . The 1990’s was a period of time when server admins bragged about how many days/hours their servers had been online. No really that was seriously a thing.
In the 1990’s there basically was 0 automation. The only people automating things were shell scripters and they were seen as genius wizards who casted spells and worked magic.
I’m not saying automation is impossible with today’s tools, but it is insanely hard. The hardest part is finding accurate information, because reading the docs will do nothing but leave you lost as hell. Most docs read like notes for those who already know how to use it, complete with lack of examples.
I can’t be the only person who is like WTF are you talking about when reading docs.
One major problem with automating with today’s tools is the fact they were designed mostly for manual use in a different time period. By this I mean most ask a series of questions that are hard as hell to answer automatically, OR EVEN FIGURE OUT THE SYNTAX TO DO SO.
This is some of the syntax I found online suggesting how to answer the questions. I borked it a little with this command, I later found out.
The above code is supposed to use Heredoc syntax which creates an infile file and feeds the info to the prompts. It doesn’t work. Not sure if plain echo “value \” would do it or not, this is the syntax I found. I did get something similar working though.
Now I must spend at least another 24 hours googling and trying and digging, because most info you find about linux is wrong.
Apparently it depends on if the script asking the questions expects answers from stdin or somewhere else, file etc. Plus I saw somewhere in the openssl docs something about echo is turned off or something? I’ll post it if I find it again.
SSL is even more fun. The docs for it are terrible. It gives you no idea of what to use how to use it etc. Purely written for the already initiated. This is a major problem I see everywhere in Technology and programming. You have people smart enough to create something, but they can’t explain how to use what they created in a way that others can just pick up and use. This causes lots of wasted human time.
It shouldn’t take days to figure out how SSH works and how to automate. Days to figure out how SSL works and automate. Days to figure out how xyz works and automate it.
This is now 2021 we need improvements to tools( especially docs) so we can more easily automate things. Our tools need to give us example files of the questions they ask and better yet a copy of how to answer them. Our tools need to be able to easily be directed to a file to read the answers from. Our tools need to focus on telling users how to use them.
Our tools need help.
I have another article coming soon on how to automate SSL/TLS certificate and csr creation with shell scripts. The same can be converted to the command line since shell scripts are just Linux commands in a file with some special syntax SOMETIMES.
I came across this issue when automating infrastructure provisioning. I needed a way to pull the repository code for my app in the provisioning scripts. I didn’t want to use the ssh keys I have setup for the entire Github account due to security. I discovered that github has the ability for you to add per repository SSH keys, called Deploy keys. The docs totally left me in the dark. I had no idea how to do any of this so I had to spend days researching. I decided to write this article to save everyone else hours of time scouring the internet trying to figure out how the hell to do this.
Why use Deploy keys?
Why would you want to use Deploy keys? When automating infrastructure provisioning you don’t want to expose your personal SSH keys. These deploy keys are going to be used only for cloning a repo, you may be able to use them for other things I didn’t research that not my problem. LOL.
SSH keys when setup correctly, allow a higher level of security than user name and password. Many people are automating by scripting a user name and password, that is BAD. Also if you don’t set a passphrase for the SSH key it won’t prompt for it in the shell terminal. Normally you want your ssh key secured with a pass phrase, but for infrastructure automation we need no pass.
I won’t cover how to automate the infrastructure that would be a series of articles. What I want to cover is how in specific to use multiple SSH keys.
The syntax is wacky as it gets. First off when you are using GIT to pull/push/clone etc. from Github, git is using SSH underneath. So in order to use multiple SSH keys you actually configure SSH not GIT, but git reads the command you type and interacts with SSH on your behalf. Totally confusing. My first few hours were filled with a lot of WTF?
First off the SSH config is stored in your users .ssh directory. On most Linux distributions that is in the user you are logged in as home directory. Basically /home/username/.ssh/ this directory will hold your SSH certs, known_hosts file, config file and others. The ssh config file is always named config and goes in the .ssh directory. If you are logged in as root it will be /root/.ssh/config. Many times when provisioning a server automatically the only user you have at first is root.
Example ssh config file should look like this
Yes it goes in a file exactly like that, no equals, no semicolons no quotes, just 1980’s YAML LOLOL. The most confusing setting above, which gets more confusing if you read the docs, is Host. Just think of it as Alias. I have no idea why it is even called Host instead of Alias. That threw me and so many others off. I kept putting the same value I had for Hostname. Hostname is the exact name of the host where your repo is, github.com for this example. Identity file is the private key file location.
Another thing to look at is git for the User. You might be able to use other names, but next I’ll show how the name part ties in.
To use the above setting to clone a repo for example you would type the following at the command line.
See the User git and the Host hostAlias. This looks so similar to the regular clone command. For example here is another one of my Github repositories a public one so you can play with this.
This is the default to clone a repo. This has a default name for User of git and a default value for Host of github.com. I haven’t experimented enough yet but I am guessing you can change the name in the configs to anything you want such as Billy and use a command like : firstname.lastname@example.org:AkashicSeer/phphtml.git
So back to the question how do you use multiple SSH/Deploy keys with Git and Github?
Each IdentityFile must be a unique ssh or deploy key, they are the same thing, both are ssh keys.
Then to clone from each for example you would use the following for example.
The format is User@Host:repo-owner-name/actual-repo.git
The dot . I am putting at the end of the clone IS AWESOME.
It tells Git to clone into the current directory and don’t use the name of the repo as the directory name. Basically just clone the damn repo into this damn folder. Without the dot it includes the repo name too. I often just want /opt/app-directory < code in that folder.
AND DON’T FORGET THE SECRET SAUCE
Now that you have multiple SSH keys you must do some special magic to let SSH know about the keys. For each key you have to tell the ssh-agent it’s name. Basically when SSH does it’s thing your SSH client has to give a list of keys to the SSH agent on the server you are contacting. GIT uses SSH so you must tell SSH where the keys are for your github accounts.
To do this on linux you start the ssh-agent then you add the keys. It is a bit of a pain. First you must start the agent, then you add the key.
#start the agent on linux like so
The value you give to ssh-add command should be the ones you used for your IdentityFile definitions. You will need to add each private key to the agent before it will work.
To test that your setup is working you can do the following and read the output. If there was a problem it will tell you, like it couldn’t find the key.
Running those commands will let you know if everything is configured properly.
BUT IT GETS FUNNER GUYS
All of the 999 things above are still not enough if you want to automate this process. If you do all of the above and try automating the process, github will prompt you for a passphrase for an ssh key. It won’t be the deploy key either, NO why do that, that would be logical and make sense. What it wants is the passhprhase to the entire account, not the deploy key.
How to fix this?
And there is still, still more, you must chmod the .ssh directory to 600 such as
chmod -R 600 /home/user/.ssh
or where ever your ssh files are stored.
You may also need to do the following.
Create a dummy instance. On this instance issue the git clone command. When it asks for the passphrase enter the passphrase for the account that OWNS THE REPO, not the deploy key passphrase which should be empty.
This will add github to known_hosts file. Now use cat to output that info and copy it. You can’t use xclip like I mention in another article, no that’s not allowed for some no brain reason. Once you copy the code from known_hosts create another file on your system called known_hosts. You will need to upload this file along with the ssh deploy keys so that you are not prompted during automated clones.
If there is some sort of openssh setting or a way to do this automatically, I haven’t found it yet.
If you would like more information on how to create the ssh deploy keys themselves, read this article I wrote.
If you want more information about ssh checkout my list of resources here
I had a nice article but somehow it got screwed to hell and back. I have no idea what I did. I will not rewrite it. This is now purely a list of resources. I really hate tinymce for this. You can’t just past text from the internet because it auto adds H4’s for some no brain reason. So you have to switch to text view to paste text, then you can switch back and add the link. So helpful.
SSH keys – basically documentation on the subject by arch linux.
For so long I have read and heard that a GREAT security feature is to force your users to change passwords every so many days/months. Linux even has a built in feature for this.
This is a really stupid idea for many, many reasons. #1 a password is a password is a password. If a hacker can guess one they can guess another. Simply forcing users to change passwords is a false feeling of security.
Another reason it is a piss poor idea is, users usually use another form of their password so they can remember it, which again solves nothing whatsoever.
Another reason this is a stupid idea is people will often forget their password.
Another reason this is a bad idea is because users usually either write their password down or store it in a regular note type program on their phone, more are using password saving software. I once used a password saver, it worked great it saved all my passwords… except for the password to unlock it. I quit using them after that. LOL
You can read more about the treacheries of forced password cycling here.
You will notice others say the same things and more here.
AWS autoscaling lets you set up groups of EC2 instances which are controlled by a load balancer. The load balancer in turn makes sure your app has the correct number of EC2 instances running at all times. If your traffic is high it adds the maximum that you set. If traffic goes down it adjust to have the minimum EC2 instances that you set.
This system is great for startups who have no idea if their app will go viral or just flop. Often they just flop. But if you are lucky and it takes off you want to be able to handle the traffic so you don’t lose users.
This is a giant subject. I’ll continue updating this article as I find and understand more about how Linux does all of this.
Most articles show how to set them in the terminal, then when you close the terminal and try to use a variable it doesn’t exist leaving you confused.
Well those posts are how to set a temporary environmental variable accessible in your terminal until it closes only. To set a lasting Environmental variable you need to use a file.
This is often needed to store sensitive api key secrets, passwords etc. You should never put any of that information in a file for your project directly, meaning don’t put it in a file that is stored in your code repository.
Developer App Variables
The idea is to create environmental variables to hold this info on your local machine, then when you put your code into production you add the necessary environmental variables to whatever controls them for your app. You can use kubernetes secrets or Hashicorp vault or if you are using something like Gitlab or Openshift continuous integration/delivery pipeline workflow will have a way to enter these values securely.
One problem you will run into with apps is how Nginx or Apache either pass or don’t pass the variables to your scripts. It can be confusing. Apache gladly passes values from the environment to our scripts, but Nginx not so much. With nginx you need to set your vars in 2 locations or run a hacky feeling 3rd party Perl script which takes your variables and passes them to Nginx. Nginx has better performance than Apache though. The story is an entire article for later though, back to the subject.
Overall Environment variables
There are many ways to set environment variables in linux and each depends on what needs to be done. Each user has their own files that set variables for them. There are also other files that set variables for programs that run on your computer like cronjob. There are also files to set variables system wide, but it depends on what is operating whether it can access it.
You can create a file with any name you want with extension .sh like example-app.sh and place it in /etc/profile.d directory.
As described in this link Linux will read all of the files in that directory and create the environmental variables.
This is basically what I have in a simple text file
You simply add a new line for each variable you want to have created for your app.
Each user on your Linux system has a .bashrc file the root, regular user etc. all have one. On Ubuntu root is disabled so settings in that .bashrc file are probably ignored. But the user you usually login as, the default name that shows in your terminal when you open it, that home directory has a .bashrc file. Located at /home/username/.bashrc If you set a variable in this file you must log out and log back into your system for them to take effect or restart your computer.
As mentioned in this article, the shell looks in several places when the terminal is invoked. Where to place it depends on what you need to do.
There are login shells, interactive shells, non interactive shells, each looks in different locations. This article has a good discussion of interactive vs non interactive. Basically interactive is when you type into the terminal, non interactive is when you run a shell script or other program that does the interaction with bash ( or other shell ) for you.
This article describes the many locations and files used to set environment variables. Not all versions of Linux will have all of these or use them, each version is different and you should read the documentation for your specific distribution. The bottom of that article gives you a better idea of where to set the variables.
For scripts like shell scripts set the environmental variables in .bashrc files. For interactive use in the terminal where the user or their commands need the variables set them in the .profile file or profile.d directory as mentioned above.