Categories
Software Development Web Development

How to fix Composer GitHub API limit (0 calls/hr) is exhausted … errors

Programming Joke.

Modern web development keeps getting easier by the day

How does a mechanic start their day? They go to work. They get their tools and start working.

How does a carpenter start their day? They go to work. They pick up their tools and begin where they left off the previous day.

How does a web developer start their day? They go to work. Turn on their computer.  Start up the needed programs. Discover updates are needed. Start to do the updates. Failures happen. Spend entire day finding every bug and quirk in every piece of software they use. Maybe their OS shit the bed. Maybe their frameworks need updating… So they spend 4 hours googling and fixing errors in the first tool. Get to the next tool, same thing, spend 2 hours fixing it. Get to the next tool and have to spend 4 hours googling, updating and fixing it. In the midst of all of that they wreck a few things and have to restart a few dozen times. Finally 10 hours later their tools are maybe ready for programming the next day.

These days as a web developer you spend more time fixing your tools than programming. Programming in Web languages is the absolute worst of the worst.

If web developers were mechanics we would be ordering new tools every day and waiting for them to arrive before we could start work because the previous days tools were either broken, stolen or worked entirely differently today than they did yesterday… and the manufacturers docs were not updated, so we have to just play with them until we figure out how they work now.

programmers tools are all like…

Now to fix the  Github issue with composer.

Side note: a long time ago I’d get these types of  errors when simply pushing my code to my own repository.

“gitHub API limit (0 calls/hr) is exhausted, could not fetch https://api.github.com/graphql. Create a GitHub OAuth token to go over the API rate limit. You can also wait until ? for the rate limit to reset.”

Aren’t we all just like..

When you get that error while trying to use composer…

Well the Composer docs say the following

Because of GitHub’s rate limits on their API it can happen that Composer prompts for authentication asking your username and password so it can go ahead with its work.

If you would prefer not to provide your GitHub credentials to Composer you can manually create a token using the procedure documented here.

Now Composer should install/update without asking for authentication.

I don’t know about you, but Composer has never once prompted me for any password. It sort of just started doing this.

Here is what I randomly got today while trying to update a Symfony project.

Calculating CHANGELOG…GitHub API limit (0 calls/hr) is exhausted, could not fetch https://api.github.com/search/issues?q=a1a70353f64f405cfbacfc4ce860af623442d6e5. Create a GitHub OAuth token to go over the API rate limit. You can also wait until ? for the rate limit to reset.

Composer never asked for any authentication password. I do notice this error it is finally prompting for a github token below this error in another message. I have no idea when this began, I used composer a few weeks ago it seems and this wasn’t an issue.

You can enter the token there and it says it saves it.  DON’T DO THAT. Stop now. Do not create and enter the token at this prompt or you will be dreadfully sorry, you will get nothing but errors about the malformed token if you do. DON’T DO IT. Hit ctrl+c to quit or close your terminal, but whatever you do, do not enter that token yet, until you have the latest version of Composer installed.

Find your version with

 
//globally installed
composer --verision

//or loally installed
php composer.phar --version

If your version composer is not the latest version then update it(2.2.9 as of this article) To do this, if you installed composer locally then you delete the composer.phar file in your project folder. If you installed it globally then you need to go to that directory probably in

/usr/local/bin/composer

where composer is the filename not a directory. Delete the composer file. You will see suggestions online that you can just run those commands and you don’t need to delete composer. That didn’t work for me. I only got permission denied errors instead. So I rage deleted the damn thing and issued the commands again and

El magico

wow magic meme
Wow like magic it works now

No more permission denied now you summonabich!!! LOL

Then rerun the commands to install composer either locally or globally. I prefer globally so I don’t have to screw around with updating each project.

Then you need to create this special token to use Composer.

You are special and need a special token!

How to create and use the token? That is explained in the Composer Docs here. You go to the github page seen below

create github token
Create github token

Don’t choose any of the Scopes, just choose to create this useless token, it doesn’t need them.

You will next see this screen.

Now you copy that token. Open your terminal and use the following command with your token where the word token is.

//locally installed composer
php composer.phar config github-oauth.github.com token
//globally installed like I suggested
composer config --global github-oauth.github.com token

And now composer should be working again.

boom how it is done baby meme
Trial and error until it works

 

Categories
Software Development Web Development

How to fix Github git keeps asking for password with ssh keys

This was a super annoying issue I have had for years. I never looked into it because I was lazy.

I setup my ssh keys like you probably did and I kept getting prompted for my password.

I kept entering it for years because the github docs are not that great and fail to mention how to fully setup the keys properly(maybe changed by now). I kept wondering why it was asking me to enter my password even with the ssh keys.

Well a few days ago August 13th 2021, github switched from allowing password ssh git push/pull to not allowing it and forcing you to use something like ssh keys. I went to push to my github repo and was told I had to use ssh keys blah blah…

I was like

spock WTF
Wait. WTF?

I thought I had setup the ssh-keys. But, what had happened was I had used the HTTPS method to pull the repo. So my git config was set to the HTTPS endpoint not the SSH endpoint. It had been a long time ago. I thought I had setup the keys, and I did. What I failed to do was switch the git remote value in the git config.

are you kidding me
Freaking seriously

Basically if you pulled a repo via https then you need to switch your git configuration to use the ssh url. I won’t make this article longer writing how to do that, here is a great, short article that explains it

You also need to know the ssh repo value which you can find in the repository under clone like this

github clone
remote repo name

I have not tried the newer GitHub CLI. I will eventually read about it and try it and update my repos to use it. I did write about using github ssh deploy keys here though.  And about using multiple deploy ssh keys here.

Categories
Software Development Web Development Web Security

How to create ssh deploy keys for github

One issue with creating SSH keys is there are so many ways to do it and no one tells you why they do what they do. A quick search will reveal almost everyone has their own way of doing it.  If you are new to ssh keys I suggest you read this article really quick.

In the github docs they tell you how to create the ssh keys like this

ssh-keygen -t ed25519 -C "your_email@example.com" 

I prefer using the following command instead :

ssh-keygen -t ed25519 -f /home/akashicseer/tests/ssh/file_name -C "akashicseer@gmail.com"

Quick command facts:

  1. ed25519 is basically the newest type of key, it is supposed to be the most secure
  2. -C is for adding a comment to the key. This helps you identify it later in places like ~/.ssh/known_hosts, ~/.ssh/authorized_keys and when you use the command ssh-add -L which prints out your public key info
  3. -t specifies the type of key. The above command tells ssh-keygen to create an ed25519 type of key more info
  4. -f /home/akashicseer/tests/ssh/file_name This tells ssh-keygen where to put the file. If you don’t specify the name then it will use a default of something like id_ed25519 for private key and id_ed25519.pub for the public key. The code above will put the files named file_name (private key) and file_name.pub ( public key) in the folder /home/akashicseer/tests/ssh/ If you don’t specify the full path to the exact folder your keys will be put into your users home directory in the default .ssh location. On Ubuntu this is /home/username/.ssh/

NOTE: for ssh deploy keys, don’t specify a passphrase when you create them or you will have to manually enter it later when Packer or whatever you use runs your provisioning code. That means you won’t be able to automate if you enter a passphrase, because it will ask the terminal user to enter the phrase to do a git clone.

There are different types of ssh keys. If you don’t add the ed25519 part then a regular ssh key of type rsa is created, this is the default type of ssh key. Basically Github documentation is showing how to create a secure type of key to use with your code deployments. You will use this key to clone your repository to your server instances.

Creating the key is only half the battle. You must decide how you will create the key, especially if you are automating deployments. When automating deployments the process becomes very complicated.

First either you create the keys you need in the instance you are creating then use the github api to add them to the proper repo. Or you create the keys on a local development computer and use something like Hashicorp Packer to upload the files to the server instance during creation. The latter is the easiest way especially in automation of the infrastructure.

If you are creating your keys locally and using Packer to upload them, you will need to login to Github and go to the deploy keys section of the specific repo to add your public key. The public key is the one that ends in .pub usually. The easiest way to copy the key value is to use xclip which I mention in this article.

If fully automating the process and creating the key on the actual instance, you must remember to eventually remove older keys. Github lets you have like 50 keys per repo max. If your repo needs to be deployed to many instances, such as a microservice structure you can contact them to get added key abilities. You could also reuse the same key, but that would require keeping the private key in a repo as well which probably isn’t a very good idea at all, since ssh keys are the same as passwords basically.

Also remember this. If you are using deploy keys only to deploy by cloning the repo, then deleting the key after the clone is perfectly fine. You only need to use this key one time to clone (aka deploy ) your code, after this it is useless. You can and probably should create new SSH keys for deployment each time and remove them from Github after you deploy, then delete them from the server instance.

Unless you plan on keeping the same instance up and trying to pull from the repo etc. That is messy. Personally I’d prefer to use Packer to create new instances when I need to and redeploy. This has the added benefit that I can upgrade the instance with security etc., test it, add my app code, test it, then swap over after migrating the database and other files. This is like creating a clean slate every time.

You will also need to know how to add the keys to ssh-agent and use them, which I cover in this article.

Here is a link to a list of resources about ssh.

Categories
Software Development Web Security

How to use Multiple ssh deploy keys with Github and Git

I came across this issue when automating infrastructure provisioning. I needed a way to pull the repository code for my app in the provisioning scripts. I didn’t want to use the ssh keys I have setup for the entire Github account due to security. I discovered that github has the ability for you to add per repository SSH keys, called Deploy keys.  The docs totally left me in the dark. I had no idea how to do any of this so I had to spend days researching. I decided to write this article to save everyone else hours of time scouring the internet trying to figure out how the hell to do this.

Why use Deploy keys?

Why would you want to use Deploy keys? When automating infrastructure provisioning you don’t want to expose your personal SSH keys. These deploy keys  are going to be used only for cloning a repo, you may be able to use them for other things I didn’t research that not my problem. LOL.

SSH keys when setup correctly, allow a higher level of security than user name and password. Many people are automating by scripting a user name and password, that is BAD. Also if you don’t set a passphrase for the SSH key it won’t prompt for it in the shell terminal. Normally you want your ssh key secured with a pass phrase, but for infrastructure automation we need no pass.

I won’t cover how to automate the infrastructure that would be a series of articles. What I want to cover is how in specific to use multiple SSH keys.

The syntax is wacky as it gets. First off when you are using GIT to pull/push/clone etc. from Github, git is using SSH underneath. So in order to use multiple SSH keys you actually configure SSH not GIT, but git reads the command you type and interacts with SSH on your behalf. Totally confusing. My first few hours were filled with a lot of WTF?

Wait. WTF?

First off the SSH config is stored in your users .ssh directory. On most Linux distributions that is in the user you are logged in as home directory. Basically /home/username/.ssh/ this directory will hold your SSH certs, known_hosts file, config file and others. The ssh config file is always named config and goes in the .ssh directory. If you are logged in as root it will be /root/.ssh/config. Many times when provisioning a server automatically the only user you have at first is root.

Example ssh config file should look like this


   Host hostAlias 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa

Yes it goes in a file exactly like that, no equals, no semicolons no quotes, just 1980’s YAML LOLOL.  The most confusing setting above, which gets more confusing if you read the docs, is Host. Just think of it as Alias. I have no idea why it is even called Host instead of Alias. That threw me and so many others off. I kept putting the same value I had for Hostname. Hostname is the exact name of the host where your repo is, github.com for this example. Identity file is the private key file location.

Another thing to look at is git for the User. You might be able to use other names, but next I’ll show how the name part ties in.

To use the above setting to clone a repo for example you would type the following at the command line.


git clone git@hostAlias:repo-owner-name/repo-name.git .

See the User git and the Host hostAlias. This looks so similar to the regular clone command. For example here is another one of my Github repositories a public one so you can play with this.

git@github.com:AkashicSeer/phphtml.git

This is the default to clone a repo. This has a default name for User of git and a default value for Host of github.com. I haven’t experimented enough yet but I am guessing you can change the name in the configs to anything you want such as Billy and use a command like :
billy@github.com:AkashicSeer/phphtml.git

So back to the question how do you use multiple SSH/Deploy keys with Git and Github?

Like this


   Host hostAlias
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa

   Host otherAlias 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa_2

   Host billy 
   User git
   Hostname github.com
   IdentityFile=/root/.ssh/id_rsa_3

Each IdentityFile must be a unique ssh or deploy key, they are the same thing, both are ssh keys.

Then to clone from each for example you would use the following for example.


git clone git@hostAlias:repo-owner-name/repo-name.git .
git clone git@otherAlias:repo-owner-name/repo-name.git .
git clone git@billy:repo-owner-name/repo-name.git .

The format is User@Host:repo-owner-name/actual-repo.git

The dot . I am putting at the end of the clone IS AWESOME.
It tells Git to clone into the current directory and don’t use the name of the repo as the directory name. Basically just clone the damn repo into this damn folder. Without the dot it includes the repo name too. I often just want /opt/app-directory < code in that folder.

AND DON’T FORGET THE SECRET SAUCE

Don’t forget the secret sauce

Now that you have multiple SSH keys you must do some special magic to let SSH know about the keys. For each key you have to tell the ssh-agent it’s name. Basically when SSH does it’s thing your SSH client has to give a list of keys to the SSH agent on the server you are contacting. GIT uses SSH so you must tell SSH where the keys are for your github accounts.

To do this on linux you start the ssh-agent then you add the keys. It is a bit of a pain. First you must start the agent, then you add the key.


#start the agent on linux like so
eval `ssh-agent`
ssh-add /path/to/your/private/key

The value you give to ssh-add command should be the ones you used for your IdentityFile definitions. You will need to add each private key to the agent before it will work.

To test that your setup is working you can do the following and read the output. If there was a problem it will tell you, like it couldn’t find the key.


ssh git@hostAlias
ssh git@otherAlias
ssh git@billy

Running those commands will let you know if everything is configured properly.

BUT IT GETS FUNNER GUYS

The fun is just beginning

All of the 999 things above are still not enough if you want to automate this process.  If you do all of the above and try automating the process, github will prompt you for a passphrase for an ssh key. It won’t be the deploy key either, NO why do that, that would be logical and make sense. What it wants is the passhprhase to the entire account, not the deploy key.

How to fix this?

And there is still, still more, you must chmod the .ssh directory to 600 such as

 chmod -R 600 /home/user/.ssh 

or where ever your ssh files are stored.

You may also need to do the following.

Create a dummy instance. On this instance issue the git clone command. When it asks for the passphrase enter the passphrase for the account that OWNS THE REPO, not the deploy key passphrase which should be empty.

This will add github to known_hosts file. Now use cat to output that info and copy it. You can’t use xclip like I mention in another article, no that’s not allowed for some no brain reason. Once you copy the code from known_hosts create another file on your system called known_hosts. You will need to upload this file along with the ssh deploy keys so that you are not prompted during automated clones.

If there is some sort of openssh setting or a way to do this automatically,  I haven’t found it yet.

If you would like more information on how to create the ssh deploy keys themselves, read this article I wrote.

If you want more information about ssh checkout my list of resources here 

A really good book I found really handy is
SSH Mastery: OpenSSH, PuTTY, Tunnels and Keys (IT Mastery Book 12)

Categories
Resources Software Development Web Development

How to push an existing project to a Github repository

While trying to figure out how to do this I found lots of bad info, info that lead to nothing but issues and bugs. I have stay so busy and do so many things I can never remember how to do anything. That is basically why this ENTIRE blog exists, my shit memory.

I found this article on DigitalOcean that works properly.

Step 1 create a repository on Github, even if the code you want to put in a repo exists, as long as it is not currently in any repo at all.

Step 2 navigate to the folder that contains the code that you want to push to the new Git/Github repo and run git init

Step 3 add the files with git add -A or git add .

Step 4 commit the files

Step 5 add the remote git origin server

Step 6 push to the remote git origin server

Or just use this script.