Categories
Uncategorized

Bash how write large amounts of text to a file

I searched and tried for days to find the answer to this. All I wanted to do was be able to basically create a file and write text to it exactly as I had entered it in a shell script. Every suggestion on the internet was fubar.

Note : to run these commands, put them in a folder like test.sh and use chmod 755 test.sh to make it executable then type bash test.sh or sh test.sh or /.test.sh

I wanted to put something like this in a file from within a shell script.



Line one text
   Line two text
   line three text
   lets talk about some text

I tried everything. I googled for days and I finally found this article  where someone had basically the same need as me. Basically the syntax to put a bunch of text into a file from shell script is this.


#!/bin/bash
testFile=/home/akashicseer/tests/test-file.txt
if [ ! -e $testFile ]; then
    touch $testFile
fi
outside_var="Some text from outside hell"

cat <<TEST  >> $testFile
Line one text
   Line two text
   line three text
NEWVAR = values
lets talk about some $NEWVAR
or not
but look at some outside text $outside_var
TEST

This uses <<HEREDOC syntax but it also redirects the input with >>. This is the oddest syntax I have ever seen so I can’t explain why it works. I would expect the redirect to be at the end of the closing TEST, but that doesn’t work. Bash heredocs are the weirdest thing other than if statements I’ve come across. Learning bash has been like traveling back in time to the 70’s or 80’s the syntax is beyond odd.

Also you will notice I tried defining a variable in the heredoc. That doesn’t work. You can copy and paste the code above and see what I mean. You don’t get errors but the variable doesn’t expand. I don’t know if it is supposed to or not. Here is a link to some info about heredoc. However what you can do is define variables outside the heredoc and use them within, see the $outside_var.

If you read the “info about heredoc” link above( in links below too) it shows this alternate syntax which works too, and makes more sense. I honestly don’t know how or why  the above ugly mess works.


#!/bin/bash
testFile=/home/akashicseer/tests/test-file.txt
if [ ! -e $testFile ]; then
    touch $testFile
fi
outside_var="Some text from outside hell"

cat > $testFile <<TEST
Line one text
   Line two text
   line three text

   but look at some outside text $outside_var
TEST

This looks a little better to me than the other version. So there you have it that is how you write lots of text to a file.  I had to search for days to figure this out so I hope this saves at least one person some time.

Version 1 heredoc syntax
Version 2 heredoc syntax

Categories
Software Development Web Security

SSH secure shell links videos and resources

I had a nice article but somehow it got screwed to hell and back. I have no idea what I did. I will not rewrite it. This is now purely a list of resources. I really hate tinymce for this. You can’t just past text from the internet because it auto adds H4’s for some no brain reason. So you have to switch to text view to paste text, then you can switch back and add the link. So helpful.

Articles

SSH keys – basically documentation on the subject by arch linux.

Tutorials point article covering ssh-keygen

Understanding the SSH Encryption and Connection Process – a really decent article going into the details of how SSH actually works underneath for anyone interested. I highly suggest reading this as it eliminates some of the questions you may have.

ssh-keygen – Generate a New SSH Key

SSH command – article about SSH on

SSH manual.– a 1990’s looking manual LOL basically the SSH documentation from what I can see.

Really good DigitalOcean article/manual about SSH

How to manage multiple SSH key pairs

SSH Keys with Multiple GitHub Accounts 

Configuring openssh for passwordless login- a guide by ubuntu about how to setup openssh to allow logins by ssh without passwords.

Host Vs HostName – you know just to be confusing

Fix: Pseudo-terminal will not be allocated because stdin is not a terminal – because SSH is stuck in 1990 and you need to give passwords and answer yes EVERYWHERE This totally makes scripting automated server deploys EXTREMELY HARD.

Securely add a host (e.g. GitHub) to the SSH known_hosts file

Managing Your SSH known_hosts Using Git

SSH known_host file syntax specification and information.


Videos

The below video gives a little history of how SSH came about. It also covers how SSH works to send and receive data.


This video goes into more depth about SSH and how to use it.

Categories
Resources Software Development

Ansible resources links videos and information

Ansible is used to provision/setup servers for your app.
Why is Ansible better than shell scripting?

Categories
Resources Software Development

Mysql resources links and information

This page contains links and videos to information about MySQL database.

More on MySQL option files in the documentation here.

Categories
Resources Software Development

Linux commands resources and links

Difference between Ubuntu apt-get autoclean, clean, autoremove commands – it is important to know the difference.

Cleaning up with apt-get – how to clean up with apt clean command

How To Upgrade Ubuntu Using Command Line?

Linux << heredoc syntax info – great link has lots of good linux information

Here is an excellent link to a shell command cheatsheet. This lists the most commonly used shell programs.

An A-Z Index of the Linux command line: bash + utilities. – about the same as the list of shell commands above.20 Shell Scripting Questions for Practice

Writing text to files from shell or shell scripts

Awesome linux software – a list/article about awesome useful linux programs
Understanding Shell Initialization Files and User Profiles in Linux

20 Shell Scripting Questions for Practiceuses opensshLinux visudo command

Linux /etc/profile file information

Understanding a little more about /etc/profile and /etc/bashrc20 Shell Scripting Questions for Practice

Why and How to Edit Your Sudoers File in Linux

LINUX DIRECTORY STRUCTURE:/ETC EXPLAINED

Tutorialspoint Linux tutorials and really good information.20 Shell Scripting Questions for Practice

How to Use Sudo and the Sudoers File – very good article

Ubuntu Environmental variables – Ubuntu Documentation link

How to prompt for input from user in a linux shell script – article 20 Shell Scripting Questions for Practicecovers two ways to prompt users for input in shell scripts.

How to prompt and read user input in a Bash shell script – A good example of a script that prompts users for input and uses it.

Bash read builtin command – this command is built into bash meaning available without installing etc.

Linux while loop explained with examples of alternate syntaxes

How to create a self signed SSL certificate 

20 Shell Scripting Questions for Practice – article showing how to prompt users for input and how to answer prompts.


Categories
Resources Web Development

How to list and install php modules on ubuntu linux

I always, always forget how to do this and it takes an hour of googling to figure this out again. Basically I often need to see not just what php modules are installed on my system, but also what can be installed.

For example you can see what is already installed and available to you on your current system you simply type php -m in your command line. This command will list all of what is installed currently on your system.

But that is not what I want and probably not what you want if you are reading this. What I want is to know what is available for installing, or better what are the exact names of the packages. For this information you need a different command.

sudo apt-cache search php7*

This command searches the apt cache for packages that contain php7* the * is a wildcard meaning anything that looks like php7 such as php7.4-mysql. Try the command above and you will get a list of all of the php modules specifically for version 7.4. Not all modules will be listed. You can use another command to see all of the available php modules even if they don’t contain 7.4 in the name you can install them.

To see all php modules available use this command.

sudo apt-cache search php*

Minus the 7 and it will return every last module that contains the word php.

Then to install anything you need you use a command like the following

sudo apt install php7.4-mysql php7.4-curl php7.4-json php7.4-cgi php7.4-xsl

And that is how you list and install php modules/extensions.

Here is an interesting article about how apt-cache works on linux
Here is a link to some more resources about this topic.

Categories
Resources Software Development Web Security

AWS autoscaling links and resources

AWS autoscaling lets you set up groups of EC2 instances which are controlled by a load balancer. The load balancer in turn makes sure your app has the correct number of EC2 instances running at all times. If your traffic is high it adds the maximum that you set. If traffic goes down it adjust to have the minimum EC2 instances that you set.

This system is great for startups who have no idea if their app will go viral or just flop. Often they just flop. But if you are lucky and it takes off you want to be able to handle the traffic so you don’t lose users.

Documentation link to AWS autoscaling

Running EC2 instances at Scale with autoscaling groups – small Ebook that walks through the whole process including using CodeDeploy.

Categories
Software Development Web Development

PHP tools for CI CD testing etc.

7 Continuous Integration Tools for PHP Laravel Developers

Categories
Resources Software Development

CI/CD resources and information.

Here is a list of Articles, videos and other information about Continuous integration, deployment etc. etc. Devops resources.

CI/CD article

Understanding the CI/CD Pipeline: What It Is, Why It Matters 

Set Up a CI/CD Pipeline on AWS 

Continuous integration vs. continuous delivery vs. continuous deployment

CICD on AWS using CodeBuild, CodeDeploy and CodePipeline

Categories
Resources Software Development

Database design info resources and links

This page is all about databases, their design, how they work, how they replicate etc.

Living without atomic clocks – article about the subject and cockroachdb

Exploiting a Natural Network Effect for Scalable,
Fine-grained Clock Synchronization – pdf link

Amazon time sync service – atomic clock ntp time syncing for AWS based systems.

STORING UTC IS NOT A SILVER BULLET – article about storing timestamps

Keeping Time With Amazon Time Sync Service

Setting the time for your Linux instance – amazon cloud

Storing Date/Times in Databases