Categories
Software Development Web Development

Symfony 5+ error logging handlers explained

This will be a super long article, this subject is way more complex than it originally sounds. As always, I’ll be updating this soon with more info and possibly more examples as I figure it out better.

First off, you are not limited to just one logger in Symfony. However, the most popular is Monolog and Symfony has built in support making it easier to implement. So that is what I cover here.

One thing you might want to use a logger for is logging specific exceptions or errors to specific locations. You may also just want to log some value in your code for debugging purposes to see what is happening or being sent to a particular method/function. These are good use cases for logging.

While most errors/exceptions already display with lots of info in the browser, you may be like me and want certain things logged so you can review the entire pattern over a period of time.

The steps required to log sound sort of simple:

  1. Install the logger
  2. Configure the logger, decide how you want it to actually work. You don’t have to configure it with yaml. Doing so makes it so that you can autowire loggers instead of having to create a new object passing values in each time which will usually be the same values.
  3. Get the logger into your class some how( this can vary depending on whether it is a controller or a service/class)
  4. Use the logger to log something like an exception.

Installing

Your project might already have monolog logger installed since Symfony uses it by default to log everything. It doesn’t matter. Running the line below won’t hurt even if it is installed already.

To install the Monolog Logger you simply open your terminal, navigate to the root of your project and type the following :

composer require symfony/monolog-bundle

That is all. Now the monolog bundle is installed and you can use it after you configure it. And that is where the fun and confusion begin. Symfony uses Monolog so it may already be installed  depending on how you created your project.

Handlers

First decide what types of loggers you want to use. You will probably see people using the stream type logger most often. This is really generic and is the default. I don’t like the stream handler because it just keeps growing infinitely.

I prefer storing different logs to different places. You can even store the logs in a database if you want. The good news is you can use more than one type of logger, and configure them however you want.

First off, here is a long list of handler types you can use with Monolog. This is going to be SUPER LONG. I’d rather just list it here than link to a file. This list is taken from the code on github

* Possible handler types and related configurations (brackets indicate optional params):
 *
 * - service:
 *   - id
 *
 * - stream:
 *   - path: string
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [file_permission]: int|null, defaults to null (0644)
 *   - [use_locking]: bool, defaults to false
 *
 * - console:
 *   - [verbosity_levels]: level => verbosity configuration
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [console_formatter_options]: array
 *
 * - firephp:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - browser_console:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - gelf:
 *   - publisher: {id: ...} or {hostname: ..., port: ..., chunk_size: ...}
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - chromephp:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - rotating_file:
 *   - path: string
 *   - [max_files]: files to keep, defaults to zero (infinite)
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [file_permission]: string|null, defaults to null
 *   - [use_locking]: bool, defaults to false
 *   - [filename_format]: string, defaults to '{filename}-{date}'
 *   - [date_format]: string, defaults to 'Y-m-d'
 *
 * - mongo:
 *   - mongo:
 *      - id: optional if host is given
 *      - host: database host name, optional if id is given
 *      - [port]: defaults to 27017
 *      - [user]: database user name
 *      - pass: mandatory only if user is present
 *      - [database]: defaults to monolog
 *      - [collection]: defaults to logs
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - elasticsearch:
 *   - elasticsearch:
 *      - id: optional if host is given
 *      - host: elastic search host name. Do not prepend with http(s)://
 *      - [port]: defaults to 9200
 *      - [transport]: transport protocol (http by default)
 *      - [user]: elastic search user name
 *      - [password]: elastic search user password
 *   - [index]: index name, defaults to monolog
 *   - [document_type]: document_type, defaults to logs
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - redis:
 *   - redis:
 *      - id: optional if host is given
 *      - host: 127.0.0.1
 *      - password: null
 *      - port: 6379
 *      - database: 0
 *      - key_name: monolog_redis
 *
 * - predis:
 *   - redis:
 *      - id: optional if host is given
 *      - host: tcp://10.0.0.1:6379
 *      - key_name: monolog_redis
 *
 * - fingers_crossed:
 *   - handler: the wrapped handler's name
 *   - [action_level|activation_strategy]: minimum level or service id to activate the handler, defaults to WARNING
 *   - [excluded_404s]: if set, the strategy will be changed to one that excludes 404s coming from URLs matching any of those patterns
 *   - [excluded_http_codes]: if set, the strategy will be changed to one that excludes specific HTTP codes (requires Symfony Monolog bridge 4.1+)
 *   - [buffer_size]: defaults to 0 (unlimited)
 *   - [stop_buffering]: bool to disable buffering once the handler has been activated, defaults to true
 *   - [passthru_level]: level name or int value for messages to always flush, disabled by default
 *   - [bubble]: bool, defaults to true
 *
 * - filter:
 *   - handler: the wrapped handler's name
 *   - [accepted_levels]: list of levels to accept
 *   - [min_level]: minimum level to accept (only used if accepted_levels not specified)
 *   - [max_level]: maximum level to accept (only used if accepted_levels not specified)
 *   - [bubble]: bool, defaults to true
 *
 * - buffer:
 *   - handler: the wrapped handler's name
 *   - [buffer_size]: defaults to 0 (unlimited)
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [flush_on_overflow]: bool, defaults to false
 *
 * - deduplication:
 *   - handler: the wrapped handler's name
 *   - [store]: The file/path where the deduplication log should be kept, defaults to %kernel.cache_dir%/monolog_dedup_*
 *   - [deduplication_level]: The minimum logging level for log records to be looked at for deduplication purposes, defaults to ERROR
 *   - [time]: The period (in seconds) during which duplicate entries should be suppressed after a given log is sent through, defaults to 60
 *   - [bubble]: bool, defaults to true
 *
 * - group:
 *   - members: the wrapped handlers by name
 *   - [bubble]: bool, defaults to true
 *
 * - whatfailuregroup:
 *   - members: the wrapped handlers by name
 *   - [bubble]: bool, defaults to true
 *
 * - syslog:
 *   - ident: string
 *   - [facility]: defaults to 'user', use any of the LOG_* facility constant but without LOG_ prefix, e.g. user for LOG_USER
 *   - [logopts]: defaults to LOG_PID
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - syslogudp:
 *   - host: syslogd host name
 *   - [port]: defaults to 514
 *   - [facility]: defaults to 'user', use any of the LOG_* facility constant but without LOG_ prefix, e.g. user for LOG_USER
 *   - [logopts]: defaults to LOG_PID
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [ident]: string, defaults to
 *
 * - swift_mailer:
 *   - from_email: optional if email_prototype is given
 *   - to_email: optional if email_prototype is given
 *   - subject: optional if email_prototype is given
 *   - [email_prototype]: service id of a message, defaults to a default message with the three fields above
 *   - [content_type]: optional if email_prototype is given, defaults to text/plain
 *   - [mailer]: mailer service, defaults to mailer
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [lazy]: use service lazy loading, bool, defaults to true
 *
 * - native_mailer:
 *   - from_email: string
 *   - to_email: string
 *   - subject: string
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [headers]: optional array containing additional headers: ['Foo: Bar', '...']
 *
 * - symfony_mailer:
 *   - from_email: optional if email_prototype is given
 *   - to_email: optional if email_prototype is given
 *   - subject: optional if email_prototype is given
 *   - [email_prototype]: service id of a message, defaults to a default message with the three fields above
 *   - [mailer]: mailer service id, defaults to mailer.mailer
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - socket:
 *   - connection_string: string
 *   - [timeout]: float
 *   - [connection_timeout]: float
 *   - [persistent]: bool
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - pushover:
 *   - token: pushover api token
 *   - user: user id or array of ids
 *   - [title]: optional title for messages, defaults to the server hostname
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [timeout]: float
 *   - [connection_timeout]: float
 *
 * - raven / sentry:
 *   - dsn: connection string
 *   - client_id: Raven client custom service id (optional)
 *   - [release]: release number of the application that will be attached to logs, defaults to null
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [auto_log_stacks]: bool, defaults to false
 *   - [environment]: string, default to null (no env specified)
 *
 * - sentry:
 *   - hub_id: Sentry hub custom service id (optional)
 *
 * - newrelic:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [app_name]: new relic app name, default null
 *
 * - hipchat:
 *   - token: hipchat api token
 *   - room: room id or name
 *   - [notify]: defaults to false
 *   - [nickname]: defaults to Monolog
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [use_ssl]: bool, defaults to true
 *   - [message_format]: text or html, defaults to text
 *   - [host]: defaults to "api.hipchat.com"
 *   - [api_version]: defaults to "v1"
 *   - [timeout]: float
 *   - [connection_timeout]: float
 *
 * - slack:
 *   - token: slack api token
 *   - channel: channel name (with starting #)
 *   - [bot_name]: defaults to Monolog
 *   - [icon_emoji]: defaults to null
 *   - [use_attachment]: bool, defaults to true
 *   - [use_short_attachment]: bool, defaults to false
 *   - [include_extra]: bool, defaults to false
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [timeout]: float
 *   - [connection_timeout]: float
 *
 * - slackwebhook:
 *   - webhook_url: slack webhook URL
 *   - channel: channel name (with starting #)
 *   - [bot_name]: defaults to Monolog
 *   - [icon_emoji]: defaults to null
 *   - [use_attachment]: bool, defaults to true
 *   - [use_short_attachment]: bool, defaults to false
 *   - [include_extra]: bool, defaults to false
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - slackbot:
 *   - team: slack team slug
 *   - token: slackbot token
 *   - channel: channel name (with starting #)
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - cube:
 *   - url: http/udp url to the cube server
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - amqp:
 *   - exchange: service id of an AMQPExchange
 *   - [exchange_name]: string, defaults to log
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - error_log:
 *   - [message_type]: int 0 or 4, defaults to 0
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - null:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - test:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - debug:
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - loggly:
 *   - token: loggly api token
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [tags]: tag names
 *
 * - logentries:
 *   - token: logentries api token
 *   - [use_ssl]: whether or not SSL encryption should be used, defaults to true
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *   - [timeout]: float
 *   - [connection_timeout]: float
 *
 * - insightops:
 *   - token: Log token supplied by InsightOps
 *   - region: Region where InsightOps account is hosted. Could be 'us' or 'eu'. Defaults to 'us'
 *   - [use_ssl]: whether or not SSL encryption should be used, defaults to true
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - flowdock:
 *   - token: flowdock api token
 *   - source: human readable identifier of the application
 *   - from_email: email address of the message sender
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - rollbar:
 *   - id: RollbarNotifier service (mandatory if token is not provided)
 *   - token: rollbar api token (skip if you provide a RollbarNotifier service id)
 *   - [config]: config values from https://github.com/rollbar/rollbar-php#configuration-reference
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * - server_log:
 *   - host: server log host. ex: 127.0.0.1:9911
 *   - [level]: level name or int value, defaults to DEBUG
 *   - [bubble]: bool, defaults to true
 *
 * All handlers can also be marked with `nested: true` to make sure they are never added explicitly to the stack
 *
 * @author Jordi Boggiano <j.boggiano@seld.be>
 * @author Christophe Coevoet <stof@notk.org>

That is a lot. Some are obvious, others you will need to google and figure out.

You can get as wild and creative as you want though.

Config location is everything

Where you place your configuration is very important. A little more hint here.

If you want to log just during Development you put the configuration inside the dev folder. /app/config/packages/dev/monolog.yaml

If you want to log only during production you put the configuration in the prod folder. /app/config/packages/prod/monolog.yaml

If you want to log in all environments you place the configuration in the main configuration folder. /app/config/packages/monolog.yaml

So if you want a specific logger that logs to a specific file to be available in all environments define it in the “packages” folder.

It isn’t too bad once you figure it out.

Here is the official documentation on how configuration works. Read it if you need more information.

An example configuration

Before getting too much further lets look at the default file I have for the configuration of monolog in the development environment.

#file name -> /config/packages/dev/monolog.yaml
monolog:
    handlers:
        main:
            type: rotating_file
            max_files: 3
            path: "%kernel.logs_dir%/%kernel.environment%.log"
            #path: php://stderr
            level: debug
            channels: ["!event"]
        # uncomment to get logging in your browser
        # you may have to allow bigger header sizes in your Web server configuration
        #firephp:
        #    type: firephp
        #    level: info
        #chromephp:
        #    type: chromephp
        #    level: info
        console:
            type: console
            process_psr_3_messages: false
            channels: ["!event", "!doctrine", "!console"]

Yours may differ, I have probably made changes to this and forgotten at this point or some changes in the Symfony code base may have happened.

After the word handlers you list each handler one tab level in.  Each handler gets it’s own configuration settings. Basically you pick a handler from the long list above, then configure it properly for your needs.

For local development I prefer the rotating_file option. Since it seems like it is the only one that you can put a limit on the number of files.

The stream option just infinitely fills a single file forever and ever. Unless you have some sort of log rotator as mentioned in the documentation.

If you have the time and urge you could even store the messages in a database then create a user interface in your admin panel to view the messages then delete them. Otherwise you will need some way of limiting the logs and log sizes.

The documentation refers to this list as a stack of handlers. You can have as many as you want, like I mentioned above.

Here is the default production configuration taken from here in the Symfony docs.

# file name -> /config/packages/prod/monolog.yaml
monolog:
    handlers:
        filter_for_errors:
            type: fingers_crossed
            # if *one* log is error or higher, pass *all* to file_log
            action_level: error
            handler: file_log

        # now passed *all* logs, but only if one log is error or higher
        file_log:
            type: stream
            path: "%kernel.logs_dir%/%kernel.environment%.log"

        # still passed *all* logs, and still only logs error or higher
        syslog_handler:
            type: syslog
            level: error

The Symfony docs say the following about production logs.

In the prod environment, logs are written to STDERR PHP stream, which works best in modern containerized applications deployed to servers without disk write permissions.

I’ll probably remove the line that logs locally to a file the file_log line. No need to infinitely fill a log I don’t read. You will need to decide what handlers you want in production and configure them.

Handler names

As you can see in the development configuration file above, there is a handler named “main”. You can give your handlers any name you want, but you need to tell Symfony what type it is with the type directive like you see in main.

main: 
  type: rotating_file 
  max_files: 3 
  path: "%kernel.logs_dir%/%kernel.environment%.log" 
  level: debug 
  channels: ["!event"]

The word “main” can be anything you desire, but notice how you must use type, then use one of the above handler names, then provide the options. Read more about handler names here in the Symfony docs. Basically you can call the handler anything even “asdfg” as long as you use the word “type” and list the type and other required information. More on this below.

Bubble?

That is the question!

One property that you will see on all of the types above is “bubble” Further digging into the github monolog documentation reveals the following about it.

Handlers also have a $bubble property which defines whether they block the record or not if they handled it. In this example, setting the MailHandler‘s $bubble argument to false means that records handled by the MailHandler will not propagate to the StreamHandler anymore.

Basically this means you need to define your loggers in the proper order if you want to have the handling stop at a particular logger for a particular situation. Otherwise each logger in your list does something each time and you might not want that. Like do you want errors to go to a log and also to the console during development?

One thing you may want to think about is where do you want your errors, exceptions and other logged data to be logged? This answer will be different for production vs development for sure. You may want to log just exceptions to a database in production, so that you can query it and show them in an admin user interface for example.

This part of the documentation speaks about logging to different locations.

Channels aka logging to multiple places

What are these channels?

The documentation speaks of channels. What are these channels? In the code above for the monolog.yaml development configuration you will see this line

 channels: ["!event", "!doctrine", "!console"]

Here in the documentation it says the following about channels

The Symfony Framework organizes log messages into channels. By default, there are several channels, including doctrine, event, security, request and more. The channel is printed in the log message and can also be used to direct different channels to different places/files.

Aha. So that is what those are. I wonder what the ! means? Does that mean it ignores those or does it mean it accepts those? Yes it does. Basically the above line would log everything except (event, doctrine and console.) Below you can see a list of all the channels.

So to clarify if you want a handler to log a message for a specific channel you can use the following syntax in the handler configuration.

channels: [security, doctrine]

If you want the handler to ignore a channel then you use this syntax in the handler configuration

channels: ["!event", "!doctrine", "!console"]

You can really get as insanely complex as you want with this logging stuff.

By the way how do you know what logging channels are available?

I prefer the following command, because these can be auto-wired :

php bin/console debug:autowiring

Scroll down and look for monolog listings. You will see something like this.(These can all be auto-wired)

console bin output

These are the channels you have.Basically whenever one of these experiences an issue Symfony sends out an event and the handlers you list decide if they want to do something about it. This way using the channels and multiple handlers you can log things exactly where you want them.  You can even log the same event/message multiple times in multiple locations.

As the docs mention, by default Symfony logs all events to the same location. It is up to you to change that to your projects needs.

Another useful command is the following which allows you to get detailed info about each, it also lists more, but not all are auto-wireable.

php bin/console debug:container monolog

That is interactive and you can choose one of the listings to get more information about it. I find the first method useful to find what I actually can access.

Creating Custom Channels

Sometimes you need to log a specific issue to a specific file or location. For this you need to create your own Channel. The documentation is really weak on this subject so lets dig deep into it.

I get tired of the foo bar BS so here I create a custom channel named “chicken” just so it sticks out for this example.

To create the channel you simple list it at the top of the monolog.yaml file like this.

monolog:
  channels: ['chicken']
  handlers:
    main:
      type: rotating_file
      max_files: 3
      path: "%kernel.logs_dir%/%kernel.environment%.log"
      #path: php://stderr
      level: debug
      channels: [ "!event" ]

A little confusing how you use that syntax in more than one location. In one it listens for channels in another it creates channels.

Wait what?

Maybe channel_list or something else to differentiate would have been better?

Placing the directive above the rest of the handler code creates the new channel or channels.  I don’t know if this is a requirement, but I know it works. To create more than one channel create strings separated by commas.

Now to check if anything happened I will use this command


php bin/console debug:autowiring

Now we have a chicken logger.

Now you can see “monolog.logger.chicken” logger that was created with that one line in the monolog.yaml configuration file.

That is how it is done.

How to use the logger?

There are two ways to use the logger, one you didn’t have to do any of this to use, the other is autowiring, which is why we did all of this for. The other way is to just initiate it and pass the values to the constructor like the documentation for Monolog shows.

Even though all that configuration seems like a pain, in the end it makes things easier by allowing us to auto-wire the logger we want. This saves a lot of having to create the same boilerplate code over and over all throughout our app.

Taking another look at the output of the bin/console debug:container command from above we can see the following

Lets now use the chickenLogger

Each line after the first monolog.logger contains what looks like a PHP variable. It is in fact a PHP variable, one you can use for type hinting. How this works is you type hint LoggerInterface, then use which variable you want and Symfony automagically loads it for you.

farth too easy meme
It feels too easy to be true

Back to the chickenLoggerExample. What I want to do with the chickenLogger is use it to log exceptions to their own exceptions log for easier viewing. This is useful in a situation like using a library to create and convert images.

class UserImageProcessor
{

    private LoggerInterface $chickenLogger;

    public function __construct(LoggerInterface $chickenLogger){
        $this->chickenLogger = $chickenLogger;
        $this->chickenLogger->debug('lets log some chickens');
    }

See in the constructor how I passed the variable $chickenLogger and now Symfony passes me the chickenLogger Channel and I can now log chickens. Logging chickens is a very handy feature in the real world.

And below is the whole monolog.yaml file I am using to log my chickens.

monolog:
  channels: ['chicken']
  handlers:
    main:
      type: rotating_file
      max_files: 3
      path: "%kernel.logs_dir%/%kernel.environment%.log"
      #path: php://stderr
      level: debug
      channels: [ "!event", "!chicken" ]

    exceptions:
      type: rotating_file
      max_files: 2
      path: "%kernel.logs_dir%/exceptions.log"
      file_permission: '765'
      level: debug
      channels: ['chicken']


    # uncomment to get logging in your browser
    # you may have to allow bigger header sizes in your Web server configuration
    #firephp:
    #    type: firephp
    #    level: info
    #chromephp:
    #    type: chromephp
    #    level: info
    console:
      type: console
      process_psr_3_messages: false
      channels: [ "!event", "!doctrine", "!console" ]

Notice I used the file permissions option. That is a handy one. See the exceptions handler is using the chicken channel. Again I could have named the “exceptions:” section anything even “chirpy-birds” symfony doesn’t care, only the directives matter.

Now when I log my chickens, the main handler section will ignore the chickens, but my exceptions section will log my chickens and so will the console. I’d have to add “!chicken” to the consoles “channels” section or I can add “bubbles: false” to the exceptions section to stop it.

Now you know!!!

Links about Loggers

Logging to Rollbar -> Helps you organize and view logs and much more.

Logging with Monolog -> A short article

Logging -> Documentation link ->overall general information

How to log messages to different files -> more from the documentations

Using the logger -> more from the docs on how to use it.

Categories
Software Development Web Development

PHP Enumeration classes vs fake enumerations what to use.

I am growing to HATE PHP ENUMS. Why? Because if you forget to call ->value you get BS errors about cannot convert object to string. It is just another un-needed step. Enums sounded cool at first but fuq the BS with calling ->value, it is entirely too easy to forget.

Enumeration classes in PHP are a new feature as of version 8.1 Fake enumerations were what we did before this new feature. A fake enumeration was just a class with constants, the syntax to refer to them was the same minus ->value. See my other article if you don’t know what fake PHP enumerations are Php Backed Enums don’t forget to call value

TLDR

Use plain classes with constants if any of the values will repeat. Use Enumerations when each value is always unique. Keep reading to find out WTF I am talking about.

WTF are you talking about?

The story

The new Enumeration classes have a limitation on them that each “case” must have an unique value, it is just how it works.
This is only a problem when you are using Enumerations more like the old fake constant enumerations, because you can’t give two cases the same value. If you are using ints you run into this issue more quickly.

Let me show an example.

I have often used this pattern to help eliminate bugs from typing, it also helps you remember what values are possible for something. Below is the values allowed for an image for some sort of upload.

class ImageDataEnum
{
    const HEIGHT = 'height';
    const SIZE_STRING = 'size';
    const IMAGE_URL = 'image_url';
    const WIDTH = 'width';
}

This way I know what values are allowed in the upload and I don’t get errors from typing.This would convert to an Enumeration perfectly.

I have dysgraphia and dyslexia so this is something I fight constantly, it is also why I hate Linux command line. Basically my brain gets the letters out of order sometimes when I type or write or leaves entire letters out or adds them and my eyes don’t see it for some odd reason sometimes. I couldn’t take notes in school because it turned into jibberish word soup.

But I also like to use them to store info that won’t change, like maximum image size. This is a use case that won’t work with PHP Enumerations, probably one of the only ones I can think of.

Bad code

You made a real mess of that code fella

This kind of code won’t work.

enum ImageSizeEnum: int
{

    case BLOG_IMAGE_HEIGHT = 600;
    case BLOG_IMAGE_WIDTH = 600;
    case TWITTER_LARGE_IMAGE_HEIGHT = 600;
    case TWITTER_LARGE_IMAGE_WIDTH = 1200;
    case PROFILE_PICTURE_HEIGHT = 180;
    case PROFILE_PICTURE_WIDTH = 180;
    case PROFILE_THUMB_HEIGHT = 55;
    case PROFILE_THUMB_WIDTH = 55;
    case USER_THUMB_HEIGHT = 150;
    case USER_THUMB_WIDTH = 150;
    case USER_IMAGE_HEIGHT = 600;
    case USER_IMAGE_WIDTH = 600;

}

Notice that while each case has a unique name, they do not have unique values. The code above will give you an error because the values for the cases is repeated. If you are using PHPStorm then it is screaming at you right now with red underlines and error warnings etc. LOLOL

PHP Storm is all like…

The docs say the following

In PHP, Enums are a special kind of object. The Enum itself is a class, and its possible cases are all single-instance objects of that class. That means Enum cases are valid objects and may be used anywhere an object may be used, including type checks

In order to do what is intended in the code above (a list of constants to be used in code later) this has to be converted back to a class with constants. It looks so similar it is crazy.

Just convert really quickly

Ok code

This code will work.

class ImageSizeEnum
{

    const BLOG_IMAGE_HEIGHT = 600;
    const BLOG_IMAGE_WIDTH = 600;
    const TWITTER_LARGE_IMAGE_HEIGHT = 600;
    const TWITTER_LARGE_IMAGE_WIDTH = 1200;
    const PROFILE_PICTURE_HEIGHT = 180;
    const PROFILE_PICTURE_WIDTH = 180;
    const PROFILE_THUMB_HEIGHT = 55;
    const PROFILE_THUMB_WIDTH = 55;
    const USER_THUMB_HEIGHT = 150;
    const USER_THUMB_WIDTH = 150;
    const USER_IMAGE_HEIGHT = 600;
    const USER_IMAGE_WIDTH = 600;

}

With the fake enum you leave off the ->value part when you need the value as noted in this article. Php Backed Enums don’t forget to call value

Now I can have constants that have the same value and PHP won’t barf errors on me.

kitten frew up meme
I make PHP barf.

You can also do something very similar in Javascript if you are interested check out this article. Faking Enumerations with Vanilla javascript

Summary

So basically if you need to use the same value with a different name, then you need the old fashioned PHP fake enumerations. If your use case has it so that every named case has a unique value then use Enumerations.

 

Categories
Software Development Web Development

Php Backed Enums don’t forget to call value

The one thing I don’t like about new PHP enums is, if you forget to call ->value you get exceptions “object can’t be converted to string” It is entirely too easy to forget to call ->value.

This means in places where I refactor code I have to remember to call ->value. Hence the article title “Php Backed Enums don’t forget to call value”

It is really easy to forget to call ->value when using these new Enums.

Well thanks to my IDE PhpStorm, I caught this error before it happened to me… in most places… most times. Nah not really I forget to call ->value all the time.

I like the concept of having an Enum class as up until version PHP 8.1 you had to create class constants and pretend they were real Enums.

Old php enums

Old PHP Enums Example

Here is how we used to do PHP Enums for forever until version 8.1

class ImageDataEnum
{
    const HEIGHT = 'height';
    const SIZE_STRING = 'size';
    const IMAGE_URL = 'image_url';
    const WIDTH = 'width';
}

And to use that in any code you simply did the following where you needed a value.

$height = ImageDataEnum::HEIGHT;

And inside $height would be the string “height” you could use this to make sure a value exists without having to spell it out every time, reducing the likelihood of bugs. This is very straight forward and easy. You can still add constants to Enum classes and use them, but it feels better using case instead.

New Enums

A backed enum looks like this. Note const is now case, class is now enum, but the rest is about the same.

enum ImageDataEnum: string
{
    case HEIGHT = 'height';
    case SIZE_STRING = 'size';
    case IMAGE_URL = 'image_url';
    case WIDTH = 'width';
}

Notice the word “string” you can use int or string but not a combination of both. Backed Enums Docs here.

Now to use the new Enums like the code above you do like this

$height = ImageDataEnum::HEIGHT->value;

Otherwise $height will be an object, one that contains  handy built in methods try() and tryFrom(). See the doc links for more info on that. You can also define your own methods.

But if you fail to call ->value and you try to use this for a string comparison you will get oopsies. You can use the IDE to hunt down all cases of the old class type enums.

//this won't work
if('height' === ImageDataEnum::HEIGHT ){
 //code to do stuff in here
}

The above will result in an error telling you the comparison is not possible. You can’t compare a string to an object.

//this will work
if('height' === ImageDataEnum::HEIGHT->value ){
 //code to do stuff in here
}

You can also call ImageDataEnum::HEIGHT->name which will return HEIGHT. So you can get the name and value using those methods.

Another nice thing about the new Enum classes is they are full on classes, you can add methods to them if you want. Like checking if a value matches any of the case values or whatever your use case is.

Enums are really handy for limiting what values can be entered by users and checking against them. Another good use I have found is creating a list of options for a Database table column.

Here is an example of a column in one of my tables that stores a medias content rating type. The system later uses this in many places to make sure that the media is of this type or that the user wants to see this type of media.

enum ContentRatingsEnum: string
{
    case EVERYONE = 'everyone';
    case MATURE = 'mature';
    case RATED_X = 'rated-x';
}

This column in a media table can only contain these values and users can only select from these values as their content preference type. This is helpful because I don’t have to type those strings in 100,000 places and when I need to change one I simply refactor with my IDE features.

Here is an excellent video that just came out about PHP ENUM’s the start of the video is anyways.

Categories
Web Development

How to change the id for a form input in Symfony 5+

If you create your forms with classes in Symfony 5+ then changing the ID of the form fields is something you are not allowed to do apparently. LOL You can add/change the class and other attributes but not the id. For some reason Symfony ONLY lets you change the id inside the template. I don’t know why.

Yes I am serious

To start with what got me even interested in trying to use attr and row_attr is when I was messing around with some of my forms, I was copying and pasting and moving parts in the template. This lead to issues as I would miss pieces or get things wrong some how. So I started trying to do the whole thing inside the FormType definition class using the methods below.

It has been pointed out that some feel it is better to define class, id etc. in the template. But as I pointed out above, I had issues with that. So below is what I found.

What doesn’t work

If you are like me then you have probably tried changing the ID by using the attr or row_attr attributes of the Type right? That seems logical right?

These two methods  are not even consistent. First off row_attr only accepts some attributes, which ones I have no idea, it ignores placeholder and id apparently. So then I tried attr, it works with placeholder but ignores ID.

Makes sense right?

It sure would be nice if those didn’t ignore the values you sent to them wouldn’t it. This problem is nearly 10 years old. Later I may look over the code update it and do a PR.

If you are like me you are using Javascript to read hidden fields from the form for various reasons. Otherwise the standard naming of ID’s works flawlessly. I had not discovered this until I had this unique use case.

What does work

So it appears the only way to do this is inconsistentYou have to do it in the form rendering code inside the template. Like this

{{ form_row(registrationForm.ajaxString, { 'id': 'ajaxString'}) }}

You can also change/add other attributes this way, but you can ONLY CHANGE THE ID THIS WAY.

Otherwise Symfony takes it upon itself to name the field for you and ignore your request.

Dictating like…

It would be much easier and consistent if I could just add the ID in the FormType definition class instead of having to add it to the template. Class and other attributes can be added/changed like this, but not id. Just a little confusing that is all.

Categories
Web Development Web Security

Our programming tools are stuck in the past

Recently I decided to start automating my infrastructure. Before this it had never occurred to me how stuck in the past our ancient tools are.

These days we have the cloud. We can fire instances up in seconds. But to do this we need ways of automating things. Tools such as SSH, SSL, GIT etc. feel stuck in the 1990’s . The 1990’s was a period of time when server admins bragged about how many days/hours their servers had been online. No really that was seriously a thing.

In the 1990’s there basically was 0 automation. The only people automating things were shell scripters and they were seen as genius wizards who casted spells and worked magic.

Automating infrastructure provisioning

I’m not saying automation is impossible with today’s tools, but it is insanely hard. The hardest part is finding accurate information, because reading the docs will do nothing but leave you lost as hell. Most docs read like notes for those who already know how to use it, complete with lack of examples.
I can’t be the only person who is like WTF are you talking about when reading docs.

This is the best you can explain this FFS?

One major problem with automating with today’s tools is the fact they were designed mostly for manual use in a different time period. By this I mean most ask a series of questions that are hard as hell to answer automatically, OR EVEN FIGURE OUT THE SYNTAX TO DO SO.

This is some of the syntax I found online suggesting how to answer the questions. I borked it a little with this command, I later found out.


openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt<< EOF
echo `#US`
echo `#Florida`
echo `#.`
echo `#.`
echo `#$this_ip`
echo `#"akashicseer@gmail.com"`
EOF

The above code is supposed to use Heredoc syntax which creates an infile file and feeds the info to the prompts. It doesn’t work. Not sure if plain echo “value \” would do it or not, this is the syntax I found. I did get something similar working though.

Now I must spend at least another 24 hours googling and trying and digging, because most info you find about linux is wrong.

Apparently it depends on if the script asking the questions expects answers from stdin or somewhere else, file etc. Plus I saw somewhere in the openssl docs something about echo is turned off or something? I’ll post it if I find it again.

You’ve got to be kidding me.

SSL is even more fun. The docs for it are terrible. It gives you no idea of what to use how to use it etc. Purely written for the already initiated. This is a major problem I see everywhere in Technology and programming. You have people smart enough to create something, but they can’t explain how to use what they created in a way that others can just pick up and use. This causes lots of wasted human time.

It shouldn’t take days to figure out how SSH works and how to automate. Days to figure out how SSL works and automate. Days to figure out how xyz works and automate it.

This is now 2021 we need improvements to tools( especially docs) so we can more easily automate things.  Our tools need to give us example files of the questions they ask and better yet a copy of how to answer them. Our tools need to be able to easily be directed to a file to read the answers from. Our tools need to focus on telling users how to use them.

Our tools need help.

Our tools need help

I have another article coming soon on how to automate SSL/TLS certificate and csr creation with shell scripts. The same can be converted to the command line since shell scripts are just Linux commands in a file with some special syntax SOMETIMES.

Categories
Resources Web Development

Golang web development resources and links

Articles

Session based authentication in Go

Categories
Resources Software Development Web Development

scala web development resources

Resources for web development with Scala

Scala tags. This library is for creating HTML from Scala. Thanks lihaoyi

ScalaCSS a library for creating CSS from Scala

Categories
Resources Software Development

functional programming resources and notes

Before you get too excited and go all functional in Scala you should know that 100% Functional Scala can be serious Memory hog because you are making copies of everything everywhere because functional means immutable data. Be careful with things such as map and flatmap they create lots of extra objects.

Avoiding loops for the sake of Functional programming strictness is a smooth brained action. When programming avoid hype, understand what you are doing, what your goals are and figure out how to get there. Never blindly ride the hype train.

This article lihaoyi points out some of the inefficiencies of Scala collections for example.

Articles

Scala Closures – article about closures in Scala

Pure functions in Scala – Article about what pure functions are in Scala

Basic Functional Programming in Scala – a short article covering some of the basics of Function programming with Scala

Videos

GOTO 2018 • Functional Programming in 40 Minutes • Russ Olsen

Categories
Resources Software Development

Microservice resources

Microservice Resources

Most of us older people have been building monoliths for most of our careers. Now BAM! a new buzzyword is born Microservices. Microservices are all about tearing apart the dreaded monolith into smaller services that all function together to perform the actions of the monolith. Microservices have many benefits over Monoliths, however they have many disadvantages as well. Microservice systems are inherently distributed. Technology has changed so much recently. In the past we had to rent entire servers or virtual private servers etc.

Now we have the cloud, we can buy server power for super cheap, my favorite host Vultr.com has hosting packages for less than $5 USD. Vultr has the most awesome hosting interface for us Developers too. No intrusive demands for a credit card, you can pay with Paypal, Bitcoin and more.

This cheap hosting combined with containerization technology such as Docker/Kubernetes, makes getting into microservices easier than ever. The problem may be deciding if you need microservices or not.

I’ve spent about two years reading and researching Microservices below I list some of the Articles, Books and Videos that helped me along the way. I list these because there is a ton of old and bad information floating around the internettens and it took some effort to get to the bottom of what is currently the best accepted practices. This also gives me the ability to look back for information I can’t quite remember where I saw it since all I do is read, day and night.

Articles

Microservices – Not A Free Lunch! — This is a good article to start with. This highlights the fact that microservices are not always appropriate for every situation and you should need them more than you want them.


Microservices guide by Martin Fowler this is an excellent first start.

This link has links to many useful articles about microservices, this is a great place to start.


Microservices communications. Why you should switch to message queues.
This article covers how message queues work in a microservices setting. How to integrate microservices with message queues and why it is better than HTTP endpoints with microservices discovery and load balancing tools.


Implementing event-based communication between microservices (integration events) This article covers using message queues for event-based microservice communication.


Service Mesh VS API Gateway VS Message Queue – when to use what?
This is a small article that covers some of the pros and cons of each communication architecture style.


Microservice pattern explained. This article covers the idea of the microservice pattern. It  covers the uses of the microservice pattern and compares it to the traditional Microservice pattern. This article glosses over communication between services. This article also covers the pros and cons of the microservice pattern. This article has good information about how to split your application into microservices, it has quite a few good links too.


 REST vs Messaging for Microservices – Which One is Best? This article has some information about microservice communication and how it has changed over time.


Message Broker or Bus – what’s the difference? This is a good article that explains the minute differences between a Message Bus and a Message Broker. One of the worst things about the Software Industry is all the gimmicky terms and buzzwords, it is enough to make you want to move on to Physics where bosons, quarks and leptons are the buzzwords.


The Saga Pattern This article describes a useful microservice communication pattern called the Saga pattern. This pattern has it’s uses but I would not use it everywhere. This pattern allows a series of events to be reversed if a condition is not met or something along the line of communication fails.


Principles for Microservices Integration

This article has some good information about transactions between microservices. This article also covers choreography vs orchestration. Orchestration leads to a highly coordinated system which can create more tight coupling between microservices because some central authority orchestrates everything it must know about everything. A choreographed system decreases coupling by using an event sourced approach allowing microservices to publish events and decide what to do when an even they are listening to occurs. See the section about Choreographed vs Orchestrated in the article.


How Integration Patterns Impact Your Microservices Architecture  This is a good article that quickly covers the most basic ways of integrating Microservices. This article also has some excellent Resource links at the bottom of the page.


Apache Kafka vs Enterprise services bus

While researching Kafka as a message broker I wanted to know what the difference between Kafka and an ESB was, this excellent article covers just that topic. It covers event sourced/based systems and how Kafka is used in this type of setup.


Positioning APIs and Services – Let’s End the Confusion

This article talks about SOA Software Oriented Architecture which was the prelude to Microservices. This article is from 2015, this is about the time the word Microservice was born. This shift from SOA to microservices has made researching microservices a pain in the ass since during the shift many things were copied over or renamed etc. It has been hard to find any sort of consensus on many of the patterns and software used to build microservices.

The article mentions how SOA uses Enterprise Service Buses (ESB) as a way to integrate services. It mentions that before ESB the services connected to each other directly. This article is actually an interesting read for those of us who have not been in the industry for 20  years, it is a great recent history lesson. Before SOA it was the monolithic architecture, the mega pains of which is what spawned the SOA culture. Once you are bitten by a monolithic pile of crap you never ever want to see or create one ever again, it is the worst horror filled nightmare one can endure.

 


Microservice Frontends

 


From Microliths to Microsystems: Jonas Bonér at QCon London

This article quickly mentions some of the bad habits of building Microservices as mini monoliths which are hard to scale. It quickly covers the basics of what makes a good microservice. This article contains many handy links worth reading. The article is a summary of a Talk from QCon Here is a link to the slides From Monoliths to Microsystems. You an find the video and summary below under the same name.


Videos

Implementing Microservices with Scala and Akka


GOTO 2014 • Microservices • Martin Fowler

Martin begins this talk by asking “What is a microservice” and compares it to a monolith giving a nice definition of Microservices.  This video gives a good overview of microservices. Here is Martin Fowlers website link to microservices, lots of good links here, the same as the top link of the page. This video contains a lot of good information, Martin talks about the past and how we ended up with Microservice architecture. In this video he explains the move away from Enterprise Service Bus in the idea of dumb pipelines and smart end points. This is a very good very short explanation of Microservices and the rules of Microservices. Martin also talks about how we should design our microservices to fail and use monitoring for self healing. He also covers Microservices vs Service Oriented Architecture (SOA). At about the 18:00 mark Martin Talks about Microservices vs Monoliths and when you should choose which architecture, this topic is covered fairly thoroughly. Personally I would say Always avoid the monolith Always unless you like pain and misery. Also this video is from 2014, things have changed especially with Container Technology like Kubernetes and Docker, which make microservices much easier now.


Greg Young – The Long Sad History of MicroServices

The talk starts at about the 9:00 minute mark. Greg makes a great point about microservices – everyone has their own definition it seems. This video makes the point that the software industry repeats everything over and over, and likes to rebuild the wheel. This video mentions smart endpoints and dumb pipes as well if you are wondering what that is.

This video talks about Software Oriented Architecture (SOA) and how we ended up with Microservices which are basically SOA done correctly, or applying best concepts to SOA. Turns out SOA was a buzzword too, just like Microservices. Microservices is basically just a new buzzwordy term for SOA.

The concepts of Microservices are very old, if you are an older developer trying to wrap your head around these microservice concepts then this video will help clarify a lot of things for you. Greg talks about many of the misunderstood concepts of Microservices like how many lines of code and microservice cohesion. This video also covers the Database per service idea vs a single database per company.
|Greg also talks about how Microservices should be asynchronous and event driven and why. At 46:00 greg talks about securing events, very interesting concept. He makes the point that microservices are not a cure for everything and it is sort of a fad.


GOTO 2016 • The Entity Microservice Trap You May Be Doing It Wrong • Fred George

I like this video because Fred takes the time to explain the history of software design and development and slowly leads up to Microservices. This video will help you avoid some pitfalls of Microservice design and help clarify what Microservices really are and how they work.


GOTO 2017 • DDD Today – Modeling Uncertainty • Vaughn Vernon

In this video Vaughn talks about some of the misconceptions we have when we first start learning about Microservices. Through a series of funny stories he covers how most people think of microservices and DDD and how we should think of the two. Microservices are distributed systems and they require modeling uncertainty. This video will also help clarify what is going on with microservices, how do they communicate with commands and events etc. I like the coverage of how the commands, events and microservices should work in this video.  This video covers a lot and will help clarify some concepts of DDD and how they are used in Microservices.


Eric Evans DDD and Microservices: At Last, Some Boundaries!

This video covers Domain Driven Design and boundaries between microservices.  Eric also covers some of the decisions that need to be made on how microservices communicate with each other.


Developing microservices with aggregates – Chris Richardson

Chris breaks down how microservices and aggregates relate in this video.  This video answers lots of questions about how Microservices are built and communicate. It also covers aggregates and how to pick what goes into an aggregate. This video does a good job explaining how Event sourcing works as well. This is actually one of the most useful videos because it covers Microservices and DDD concepts at a very high view level, giving you an overall idea of what everything means.  This video really helps tie many of the Microservice and DDD concepts together and will give you aha moments. microservices.io mentioned in the video half way through.


The Enterprise Architects Intro To Microservices Part 1 Kevin Webber

This video starts by covering some of the history of software design and how the industry arrived at Microservices. Kevin then covers how users expectations have changed since the internet was born and how these days people expect things nearly immediately. This video is an excellent explanation of why the microservice architecture is so important in the Enterprise world.

Kevin soon covers the history of SOA, how and why it was a failure because ESB was a single point of failure. This gives good coverage of Enterprise service bus failings. This video gives a great explanation of what a microservice is with a nice diagram. This is an absolutely excellent explanation of the Microservices Architecture or MSA.


The Enterprise Architects Intro To Microservices Part 2 – Jonas Boner

This is the second video in the Lightbend Microservice series. The talk begins at about the 3:00 mark, before is an introduction. This video starts with the differences between hardware in the past compared to today. Jonas talks about how microservices allow for better isolation between services and why this is good for preventing entire cascading system failure when one service fails. He then compares Microservices to the Titanic and how microservices allow natural bulkheading because of their isolation by nature. This isolation allows for self healing microservices.

Why Asynchronous communication?(15:00 mark)

Jonas then talks about asynchronous communication, why synchronous is bad and asynchronous is better. If you are wondering why everyone is saying you should use asynchronous then this video gives an excellent explanation with a nice diagram. You shouldn’t always use asynchronous communication, you can also use a Message Broker and pass messages.

What is microservice state? (~26:00 mark)

Jonas talks about microservices should own their state, they should own their data or database that is. This is one of the most important points of Microservices. One microservice should not reach into another microservices database either to write or read. He talks about the fact that there is no such thing as stateless architectures (~28:00 mark) scammy buzzword.

Bounded Contexts (~29:00)Jonas talks about bounded contexts in relation to microservices. Event Logging and Event Sourcing explained(~31:00 mark)


The Enterprise Architects Intro To Microservices Part 3 – Jonas Boner

This is the third video in the Enterprise Architects series. This video starts out by giving a good description of microservices and how they should work. Jonas talks about how microservices are distributed systems by nature and the sort of things that need to be considered when designing your architecture. Jonas makes the point that no matter what in a microservices system you will be dealing with stale data and inconsistencies and that is just the way it is. Inside data vs Outside data (~10:00 mark) this is a good concept to understand. One microservice communicates to another it wants something done via a command.  Commands should have a side effect. The challenges of microservices (~11:30 mark ) Service discovery (~13:00 mark) is covered. Anti-corruption layer discussed (~16:15 mark) Api gateway discussed (~17:00 mark) Communication challenges discussed (~19:00 mark) Integrating with other systems discussed (~21:00 mark ) Circuit breakers explained (~22:00 mark) Back pressure explained (~24:32 mark ) Securing microservices discussed quickly (~26:00 mark ) Saga Pattern discussed (~35:45 mark) More about Saga pattern here.

Books listed in video

Reactive Microservices Architecture: free O’Reilly report by Lightbend CTO Jonas Bonér

Developing Reactive Microservices: free O’Reilly mini-book by Java Champion Markus Eisele


GOTO 2019 • 3 Common Pitfalls in Microservice Integration & How to Avoid Them • Bernd Rücker

Bernd is an entertaining person and he is quite funny in this video at times. Here is a link to the article mentioned in the video. He mentions in this video one of the biggest problems with microservices is they are distributed systems and many developers don’t know how distributed systems work. Developers building distributed systems should plan for failure (3:00). At about 5:30 Bernd mentions using REST for service communication and the issues involved. At about 8:30 he talks about circuit breakers with REST communication. Handling service state ~15:30. Workflow engines ~18:00 . Idempotency ~24:30 and why it is important. Asynchronous communication ~32:30. Message Bus 34:30 and monitoring dead messages. Distributed transactions ~38:30 Saga pattern mentioned ~41:00


Lost in Transaction – Strategies to deal with consistency in distributed systems (QCon London 2019)Bernd Rücker

Bernd talks about transactions, as in when you have two or more things that depend on each other. This is the video mentioned in the video above 3 Common Pitfalls… At ~3:30 he mentions Two Phase Commit (2pc) and why it doesn’t work with distributed transactions. Idempotentcy and why it is important ~10:00 and it’s basic implementation. Stateful retry strategy ~15:45 describes retry strategies with a state machine. An example of using a message queue pattern ~22:30 Saga pattern described ~27:30 and how it got it’s name, I hate the name. Choreography vs Orchestration event-driven pattern ~29:00 He goes on to talk about why fully 100% event driven systems are not the best way to build microservices. At ~35:00 he starts to talk about alternatives involving orchestration using commands. Orchestration with Saga workflows and BPMN(business process model and notation) pattern ~36:30

 


Distributed Sagas: A Protocol for Coordinating Microservices – Caitie McCaffrey – JOTB17

Sagas are a way to orchestrate transactions between microservices that need to either all succeed or all fail. This video was mentioned in the video above “Lost in Transaction”. Caitie speaks a little too fast, you may need to adjust the video speed down some. This is a great video that does a good job of explaining Sagas and what a Distributed Saga is . She covers what 2pc ( two phase commits) are and why they don’t work because they don’t scale well. She then goes on to talk about distributed Sagas. A saga is a way to reverse transactions when one fails in a series. The talk about Distributed Sagas starts at ~15:00, this is the definition. How to define a distributed saga ~ 21:30. Distributed Saga log ~23:00, basically using a log to move a multi-transaction process forward or reverse it in error. Failures in Distributed Sagas ~27:00, explains how to perform a Saga Rollback. Each of your microservices need a way to reverse transactions.


Avoiding Microservice Megadisasters – Jimmy Bogard

There are some things to avoid when designing a Microservice Architecture (MSA). In this excellent video Jimmy highlights some of the “do not do’s” of Microservice design. He does this by telling a story of history to help us understand how we got to where we are and the mistakes we made along the way. At ~10:00 he starts to cover API hell, and how it is created by microservices calling microservices calling microservices via HTTP requests tying up the whole network. This is naturally how most of us think of structuring microservice communication until we dig much deeper and find videos like this that explain why it is bad. At ~14:00 Jimmy explains how we can be naive as developers by only locally testing our microservices with mock outs and never tapping into a live system for testing, this is a very important point. At ~15:00 he points out an older Microservice design that failed. So many of the ideas on how to design Microservice architectures is wrong even though they seem logical. At ~17:00 he points out what a mega failure the management of the IT industry is, basically how not to run a company. At ~19:45 he gives the definition of a Microservice. At ~20:30 he defines and explains what a DDD bounded context is and how it relates to Microservice Architectures (MSA) At ~24:00 Jimmy gets to data duplication and why it is needed in Microservice architectures. At ~27:00 he explains Service Dependency Inversion and why it is needed in a Microservice Architecture. This dependency inversion is a great design for searches especially. At ~38:00 he again points out how not to run a business, especially not a software business. Idiot managers. At ~39:00 Jimmy’s law.


GOTO 2017 • The Seven (More) Deadly Sins of Microservices • Daniel Bryant

Link to the slides used in the video.

Continuous Delivery with Containers:The Good, the Bad, and the Ugly free book by Daniel Bryant

Containerizing Continuous Delivery in Java – another free book by Daniel Bryant

Daniel starts out by telling a short story about microservices, a very good talk about some of the history. At ~6:30 he makes an excellent point about moving to Microservices, you really need to plan your architecture well. You need to think about what technologies you will need in your architecture to make it function, the more you plan the better. At ~8:45 to containerize or not, he talks about Docker and containerization. GRPC ~12:00 quickly mentioned. Service Mesh’s covered ~14:00 More about RPC ~15:00 comparing it to event sourcing Link to the article mentioned at ~15:30  From Microliths to Microsystems: Jonas Bonér at QCon London   Next up ESB ( Enterprise Service Buses) ~16:30 these are used in SOA (Service Oriented Architectures) which were popular before Microservices became a word. Microservices are a continuation of the SOA concepts, taking the good leaving the bad basically. Non functional requirements ~23:00 He makes another good point of things you need to consider upfront when designing your microservice system. At ~26:00 he mentions Performance and Load testing with Gatling, JMeter and Flood.io. He then covers security testing ~27:15 Owasp Dependency Check, Docker Bench etc. The discussion about Wrath begins ~28:30 He talks about Devops starting ~30:30 some useful information about books there. PAAS Platforms as a Service ~35:00 Data Models ~36:00 Domain Driven Design DDD mentioned ~36:30 Context Mapping and Event Storming ~38:00 Data bases and Data Stores, RDBMS, NoSQL etc. ~39:00 At ~40:00 he mentions testing At ~41:30 he mentions API Simulation and Service Virtualization this is a very interesting concept. At ~43:00 Service Virtualization software information listed.


From Microliths to Microsystems GOTO 2017 – Jonas Boner

This video is mentioned in articles and videos above. Jonas starts the video with a discussion about Monoliths and how they don’t work. This is an excellent video that points out some bad habits to avoid, things that seem intuitive in Microservice systems is actually the wrong way to do it. At ~5:00 he starts talking about the most common way we developers begin creating microservices. At ~6:00 he begins to define what a Microlith is. Reactive Programing vs Reactive Systems ~16:30 Async discussed ~18:30 Back pressure ~20:00 Microservices come as Systems ~23:00 Separating the Stateless behavior from the Stateful entity ~23:30 Asynchronous Message Passing ~28:00 An excellent book on this subject is Reactive Messaging Patterns with the Actor Model: Applications and Integration in Scala and Akka  by Vaugh Vernon .  Thinking in terms of Consistent Boundaries ~34:00 Inside Data vs Outside Data ~35:00 DDD and Events first design ~36:00 Defining Bounded contexts by Events ~37:00 Event Storming ~38:00 Event Logging ~41:30 CQRS and Event Sourcing ~43:30 decouple read from write. What about Transactions? ~47:30


Microservices, Kubernetes, and Application Modernization Done Right

This video covers multiple subjects, many listed individually above. This is sort of like tying it all together. It talks about the mistakes of the SOA era.

Categories
Resources Web Development

Javascript resources

javascriptweekly – keep up to date on the latest javascript news

How to deal with dirty side effects in your pure functional JavaScript

async/await is just the do-notation of the Promise monad

Do we really need classes in JavaScript after all?

The cost of javascript. Javascript is often misused and abused or poor practices are used. This article covers some of the things you can do to lighten your apps javascript load and clean up your code a little.

Should You Use Classes in JavaScript?

A re-introduction to JavaScript (JS tutorial)  – because developers couldn’t be bothered with the hard task of learning actual Javascript syntax we had to pollute the syntax even more with Classes etc.

Function expressions explained

Arrow function expressions explained – examples and explanation

Mocking is a code smell – interesting article about how you can get https://krasimirtsonev.com/blog/article/javascript-managing-events-dispatch-listenout of control with testing and dependency injection. It is about javascript but applies to other languages.

Eradicating memory leaks in Javascript – A very good article for anyone to who creates Apps with Javascript.

Javascript managing events

Javascript let vs var vs const – a good article explaining the differences.

Javascript free books and resources – nice article with links to various sources.

javascript.info – a really awesome site that teaches everything you need to know about modern Javascript.