Categories
Uncategorized

Bash how write large amounts of text to a file

I searched and tried for days to find the answer to this. All I wanted to do was be able to basically create a file and write text to it exactly as I had entered it in a shell script. Every suggestion on the internet was fubar.

Note : to run these commands, put them in a folder like test.sh and use chmod 755 test.sh to make it executable then type bash test.sh or sh test.sh or /.test.sh

I wanted to put something like this in a file from within a shell script.



Line one text
   Line two text
   line three text
   lets talk about some text

I tried everything. I googled for days and I finally found this article  where someone had basically the same need as me. Basically the syntax to put a bunch of text into a file from shell script is this.


#!/bin/bash
testFile=/home/akashicseer/tests/test-file.txt
if [ ! -e $testFile ]; then
    touch $testFile
fi
outside_var="Some text from outside hell"

cat <<TEST  >> $testFile
Line one text
   Line two text
   line three text
NEWVAR = values
lets talk about some $NEWVAR
or not
but look at some outside text $outside_var
TEST

This uses <<HEREDOC syntax but it also redirects the input with >>. This is the oddest syntax I have ever seen so I can’t explain why it works. I would expect the redirect to be at the end of the closing TEST, but that doesn’t work. Bash heredocs are the weirdest thing other than if statements I’ve come across. Learning bash has been like traveling back in time to the 70’s or 80’s the syntax is beyond odd.

Also you will notice I tried defining a variable in the heredoc. That doesn’t work. You can copy and paste the code above and see what I mean. You don’t get errors but the variable doesn’t expand. I don’t know if it is supposed to or not. Here is a link to some info about heredoc. However what you can do is define variables outside the heredoc and use them within, see the $outside_var.

If you read the “info about heredoc” link above( in links below too) it shows this alternate syntax which works too, and makes more sense. I honestly don’t know how or why  the above ugly mess works.


#!/bin/bash
testFile=/home/akashicseer/tests/test-file.txt
if [ ! -e $testFile ]; then
    touch $testFile
fi
outside_var="Some text from outside hell"

cat > $testFile <<TEST
Line one text
   Line two text
   line three text

   but look at some outside text $outside_var
TEST

This looks a little better to me than the other version. So there you have it that is how you write lots of text to a file.  I had to search for days to figure this out so I hope this saves at least one person some time.

Version 1 heredoc syntax
Version 2 heredoc syntax

Categories
Resources Software Development Web Development Web Security

SSL links, videos and other resources.

SSL is a very important subject. All websites/apps should be using it. However the docs will leave you scratching your head saying WTF? So I am creating this long list of resources for anyone else who ever has to learn how to use it.

Articles

First here is a link to the docs – this will cause confusion as nothing tells you how to use the pieces together.  So it is like looking into a box of legos and knowing it builds something but you don’t even have a picture as a hint. The best you can do is use the pieces to build something that doesn’t even resemble the original creation.

OpenSSL quick reference by digicert – a very brief introduction to SSL and how it works

SSL Certificate Security Glossary – list of terms and definitions

How to create a CSR with openssl – shows some of the syntax for the -config file option.

Docs explaining the config file found in the article above bout how to create a csr with openssl

SSL Basics: What is a Certificate Signing Request (CSR)? – Exactly WTF is a CSR

Openssl config file example – openssl docs are pure 100% utter shit. I had to dig and dig and google and dig for days to find this.


Videos

Categories
Web Development Web Security

Our programming tools are stuck in the past

Recently I decided to start automating my infrastructure. Before this it had never occurred to me how stuck in the past our ancient tools are.

These days we have the cloud. We can fire instances up in seconds. But to do this we need ways of automating things. Tools such as SSH, SSL, GIT etc. feel stuck in the 1990’s . The 1990’s was a period of time when server admins bragged about how many days/hours their servers had been online. No really that was seriously a thing.

In the 1990’s there basically was 0 automation. The only people automating things were shell scripters and they were seen as genius wizards who casted spells and worked magic.

Automating infrastructure provisioning

I’m not saying automation is impossible with today’s tools, but it is insanely hard. The hardest part is finding accurate information, because reading the docs will do nothing but leave you lost as hell. Most docs read like notes for those who already know how to use it, complete with lack of examples.
I can’t be the only person who is like WTF are you talking about when reading docs.

This is the best you can explain this FFS?

One major problem with automating with today’s tools is the fact they were designed mostly for manual use in a different time period. By this I mean most ask a series of questions that are hard as hell to answer automatically, OR EVEN FIGURE OUT THE SYNTAX TO DO SO.

This is some of the syntax I found online suggesting how to answer the questions. I borked it a little with this command, I later found out.


openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt<< EOF
echo `#US`
echo `#Florida`
echo `#.`
echo `#.`
echo `#$this_ip`
echo `#"akashicseer@gmail.com"`
EOF

The above code is supposed to use Heredoc syntax which creates an infile file and feeds the info to the prompts. It doesn’t work. Not sure if plain echo “value \” would do it or not, this is the syntax I found. I did get something similar working though.

Now I must spend at least another 24 hours googling and trying and digging, because most info you find about linux is wrong.

Apparently it depends on if the script asking the questions expects answers from stdin or somewhere else, file etc. Plus I saw somewhere in the openssl docs something about echo is turned off or something? I’ll post it if I find it again.

You’ve got to be kidding me.

SSL is even more fun. The docs for it are terrible. It gives you no idea of what to use how to use it etc. Purely written for the already initiated. This is a major problem I see everywhere in Technology and programming. You have people smart enough to create something, but they can’t explain how to use what they created in a way that others can just pick up and use. This causes lots of wasted human time.

It shouldn’t take days to figure out how SSH works and how to automate. Days to figure out how SSL works and automate. Days to figure out how xyz works and automate it.

This is now 2021 we need improvements to tools( especially docs) so we can more easily automate things.  Our tools need to give us example files of the questions they ask and better yet a copy of how to answer them. Our tools need to be able to easily be directed to a file to read the answers from. Our tools need to focus on telling users how to use them.

Our tools need help.

Our tools need help

I have another article coming soon on how to automate SSL/TLS certificate and csr creation with shell scripts. The same can be converted to the command line since shell scripts are just Linux commands in a file with some special syntax SOMETIMES.

Categories
Software Development Web Security

How to use Multiple ssh deploy keys with Github and Git

I came across this issue when automating infrastructure provisioning. I needed a way to pull the repository code for my app in the provisioning scripts. I didn’t want to use the ssh keys I have setup for the entire Github account due to security. I discovered that github has the ability for you to add per repository SSH keys, called Deploy keys.  The docs totally left me in the dark. I had no idea how to do any of this so I had to spend days researching. I decided to write this article to save everyone else hours of time scouring the internet trying to figure out how the hell to do this.

Why use Deploy keys?

Why would you want to use Deploy keys? When automating infrastructure provisioning you don’t want to expose your personal SSH keys. These deploy keys  are going to be used only for cloning a repo, you may be able to use them for other things I didn’t research that not my problem. LOL.

SSH keys when setup correctly, allow a higher level of security than user name and password. Many people are automating by scripting a user name and password, that is BAD. Also if you don’t set a passphrase for the SSH key it won’t prompt for it in the shell terminal. Normally you want your ssh key secured with a pass phrase, but for infrastructure automation we need no pass.

I won’t cover how to automate the infrastructure that would be a series of articles. What I want to cover is how in specific to use multiple SSH keys.

The syntax is wacky as it gets. First off when you are using GIT to pull/push/clone etc. from Github, git is using SSH underneath. So in order to use multiple SSH keys you actually configure SSH not GIT, but git reads the command you type and interacts with SSH on your behalf. Totally confusing. My first few hours were filled with a lot of WTF?

Wait. WTF?

First off the SSH config is stored in your users .ssh directory. On most Linux distributions that is in the user you are logged in as home directory. Basically /home/username/.ssh/ this directory will hold your SSH certs, known_hosts file, config file and others. The ssh config file is always named config and goes in the .ssh directory. If you are logged in as root it will be /root/.ssh/config. Many times when provisioning a server automatically the only user you have at first is root.

Example ssh config file should look like this


   Host hostAlias 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa

Yes it goes in a file exactly like that, no equals, no semicolons no quotes, just 1980’s YAML LOLOL.  The most confusing setting above, which gets more confusing if you read the docs, is Host. Just think of it as Alias. I have no idea why it is even called Host instead of Alias. That threw me and so many others off. I kept putting the same value I had for Hostname. Hostname is the exact name of the host where your repo is, github.com for this example. Identity file is the private key file location.

Another thing to look at is git for the User. You might be able to use other names, but next I’ll show how the name part ties in.

To use the above setting to clone a repo for example you would type the following at the command line.


git clone git@hostAlias:repo-owner-name/repo-name.git .

See the User git and the Host hostAlias. This looks so similar to the regular clone command. For example here is another one of my Github repositories a public one so you can play with this.

git@github.com:AkashicSeer/phphtml.git

This is the default to clone a repo. This has a default name for User of git and a default value for Host of github.com. I haven’t experimented enough yet but I am guessing you can change the name in the configs to anything you want such as Billy and use a command like :
billy@github.com:AkashicSeer/phphtml.git

So back to the question how do you use multiple SSH/Deploy keys with Git and Github?

Like this


   Host hostAlias
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa

   Host otherAlias 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa_2

   Host billy 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa_3

Each IdentityFile must be a unique ssh or deploy key, they are the same thing, both are ssh keys.

Then to clone from each for example you would use the following for example.


git clone git@hostAlias:repo-owner-name/repo-name.git .
git clone git@otherAlias:repo-owner-name/repo-name.git .
git clone git@billy:repo-owner-name/repo-name.git .

The format is User@Host:repo-owner-name/actual-repo.git

The dot . I am putting at the end of the clone IS AWESOME.
It tells Git to clone into the current directory and don’t use the name of the repo as the directory name. Basically just clone the damn repo into this damn folder. Without the dot it includes the repo name too. I often just want /opt/app-directory < code in that folder.

AND DON’T FORGET THE SECRET SAUCE

Don’t forget the secret sauce

Now that you have multiple SSH keys you must do some special magic to let SSH know about the keys. For each key you have to tell the ssh-agent it’s name. Basically when SSH does it’s thing your SSH client has to give a list of keys to the SSH agent on the server you are contacting. GIT uses SSH so you must tell SSH where the keys are for your github accounts.

To do this on linux you start the ssh-agent then you add the keys. It is a bit of a pain. First you must start the agent, then you add the key.


#start the agent on linux like so
eval `ssh-agent`
ssh-add /path/to/your/private/key

The value you give to ssh-add command should be the ones you used for your IdentityFile definitions. You will need to add each private key to the agent before it will work.

To test that your setup is working you can do the following and read the output. If there was a problem it will tell you, like it couldn’t find the key.


ssh git@hostAlias
ssh git@otherAlias
ssh git@billy

Running those commands will let you know if everything is configured properly.

BUT IT GETS FUNNER GUYS

The fun is just beginning

All of the 999 things above are still not enough if you want to automate this process.  If you do all of the above and try automating the process, github will prompt you for a passphrase for an ssh key. It won’t be the deploy key either, NO why do that, that would be logical and make sense. What it wants is the passhprhase to the entire account, not the deploy key.

How to fix this?

And there is still, still more, you must chmod the .ssh directory to 600 such as

 chmod -R 600 /home/user/.ssh 

or where ever your ssh files are stored.

You may also need to do the following.

Create a dummy instance. On this instance issue the git clone command. When it asks for the passphrase enter the passphrase for the account that OWNS THE REPO, not the deploy key passphrase which should be empty.

This will add github to known_hosts file. Now use cat to output that info and copy it. You can’t use xclip like I mention in another article, no that’s not allowed for some no brain reason. Once you copy the code from known_hosts create another file on your system called known_hosts. You will need to upload this file along with the ssh deploy keys so that you are not prompted during automated clones.

If there is some sort of openssh setting or a way to do this automatically,  I haven’t found it yet.

If you would like more information on how to create the ssh deploy keys themselves, read this article I wrote.

If you want more information about ssh checkout my list of resources here 

A really good book I found really handy is
SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys (IT Mastery Book 12)

Categories
Resources Software Development

How to make linux shell scripts wait for a command to finish before running another

I am writing this so when other people google how to do it, they have something to find to save them time.

For days I tried to figure out how to make sure a command finished before another was run. I couldn’t find any information anywhere. If you are like me you may be thinking ( or wondering if ) that the shell just zooms through the commands you put in a script file without waiting for each to finish. It seems like this because everything is rushing by so quickly you can’t read it.

For days I was running scripts to install and configure my servers and it kept hanging so bad I couldn’t even ping the server.

I was running the following for example

sudo apt-get update
sudo apt-get dist-upgrade -y
sudo apt-get reboot

And since my server instances were hard freezing right up, I figured it must be zooming right on through causing an error.

Well come to find out after much research and someone on twitter finally confirmed to me that the shell automatically waits for each command to finish before executing the next. The shell doesn’t automatically run all commands encountered at the same time.

Now I must figure out what is actually locking my server instances up.

now you know
Categories
Software Development Web Security

SSH secure shell links videos and resources

I had a nice article but somehow it got screwed to hell and back. I have no idea what I did. I will not rewrite it. This is now purely a list of resources. I really hate tinymce for this. You can’t just past text from the internet because it auto adds H4’s for some no brain reason. So you have to switch to text view to paste text, then you can switch back and add the link. So helpful.

Articles

SSH keys – basically documentation on the subject by arch linux.

Tutorials point article covering ssh-keygen

Understanding the SSH Encryption and Connection Process – a really decent article going into the details of how SSH actually works underneath for anyone interested. I highly suggest reading this as it eliminates some of the questions you may have.

ssh-keygen – Generate a New SSH Key

SSH command – article about SSH on

SSH manual.– a 1990’s looking manual LOL basically the SSH documentation from what I can see.

Really good DigitalOcean article/manual about SSH

How to manage multiple SSH key pairs

SSH Keys with Multiple GitHub Accounts 

Configuring openssh for passwordless login- a guide by ubuntu about how to setup openssh to allow logins by ssh without passwords.

Host Vs HostName – you know just to be confusing

Fix: Pseudo-terminal will not be allocated because stdin is not a terminal – because SSH is stuck in 1990 and you need to give passwords and answer yes EVERYWHERE This totally makes scripting automated server deploys EXTREMELY HARD.

Securely add a host (e.g. GitHub) to the SSH known_hosts file

Managing Your SSH known_hosts Using Git

SSH known_host file syntax specification and information.


Videos

The below video gives a little history of how SSH came about. It also covers how SSH works to send and receive data.


This video goes into more depth about SSH and how to use it.

Categories
Resources Software Development

What does a dollar sign followed by a square bracket $[…] mean in bash?

I saw something similar to this in some code in one of my books
var=$[ $var1 - $var2 ]

I wanted to know what it did and why it was used. I’m a perfectionist with OCD.

Turns out it is deprecated from the BASH language.
Originally $[] was used to do math in Bash scripts to do Math known as arithmetic expansion.

So the new way in BASH is to use the following syntax.

var=$(( $var1 - $var2))

Basically what this syntax does is it allows you to do math more easily. Without the above syntax you have to escape certain characters like >< With the above syntax you can basically do math without escaping plus youcan use post-increment $var++, post-decrement $var– , logical and &&, logical or || bitwise math etc. It really helps you out.

Further links, resources and information

More info on stackexchange Same as the link above

Another good source of info about the (()) syntax as used in if and while statements is found in the book Linux Command Line and Shell Scripting Bible.  starting on page 325 If you don’t own the book I highly suggest it. I’ve found one errata so far and that is what this post is about.

More links and resources to BASH scripting

More links and linux resources

Categories
Software Development Web Development

Linux bash scripting command substitution aka $(command)

Linux has this syntax that looks like so:
$(command)

This is called command substitution. This allows you to get information about the execution of the command instead of having it it directed to STDOUT aka the terminal screen as usual.

That is very useful actually because you can run a command and store the output in a variable and use it anywhere you want later.

A simple example you can easily play with:

DIR_LISTINGS=$(ls -al)
echo $DIR_LISTINGS

This is so simple you don’t even have to add it to a script you can run it straight in your terminal in any directory you user owns.

More information can be found in this excellent book Linux command line and shell scripting bible page 277

More info about command substitution.

Bash manual reference.

Categories
Resources Software Development

Debian Ubuntu Linux debconf resources and information

The debconf programmers tutorial – excellent tutorial on what debconf is and how to use it.

debconf documentation

Using debconf to configure a system – article about using debconf, gives a little more explanation of what it is.

Installing MySQL with debconf – good article

Categories
Software Development Uncategorized

Linux xclip command makes command line life easier

I discovered a new tool today while adding my ssh keys to Github, something called xclip.

The xclip command makes it easy to capture output to the clipboard so you can paste it to another location like into a browser or word file etc.

The github docs above have you use it to copy your RSA key and save it for pushing your repository so you don’t have to supply a user name and password.

xclip -selection clipboard < ~/.ssh/id_ed25519.pub

The above tells xclip to put the contents of id_ed25519.pub into the clipboard. You can then use ctrl + v to paste it anywhere you need it.

Here is more info and examples on how to use xclip

Link to some man page  type info about the options it accepts as arguments.

Another link to similar info about xclip it’s options etc.