Categories
AWS

How to AWS mount EBS volume on ec2 instance ?

#1 – sudo lsblk -f

#2 – [ec2-user ~]$ sudo mount /dev/xvdf /data

Make an Amazon EBS volume available for use on Linux

PDFKindleRSS

After you attach an Amazon EBS volume to your instance, it is exposed as a block device. You can format the volume with any file system and then mount it. After you make the EBS volume available for use, you can access it in the same ways that you access any other volume. Any data written to this file system is written to the EBS volume and is transparent to applications using the device.

You can take snapshots of your EBS volume for backup purposes or to use as a baseline when you create another volume. For more information, see Amazon EBS snapshots.

You can get directions for volumes on a Windows instance from Make a volume available for use on Windows in the Amazon EC2 User Guide for Windows Instances.

Format and mount an attached volume

Suppose that you have an EC2 instance with an EBS volume for the root device, /dev/xvda, and that you have just attached an empty EBS volume to the instance using /dev/sdf. Use the following procedure to make the newly attached volume available for use.

To format and mount an EBS volume on Linux

  1. Connect to your instance using SSH. For more information, see Connect to your Linux instance.
  2. The device could be attached to the instance with a different device name than you specified in the block device mapping. For more information, see Name devices on Linux instances. Use the lsblk command to view your available disk devices and their mount points (if applicable) to help you determine the correct device name to use. The output of lsblk removes the /dev/ prefix from full device paths.The following is example output for an instance built on the Nitro System, which exposes EBS volumes as NVMe block devices. The root device is /dev/nvme0n1. The attached volume is /dev/nvme1n1, which is not yet mounted.[ec2-user ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme1n1 259:0 0 10G 0 disk nvme0n1 259:1 0 8G 0 disk -nvme0n1p1 259:2 0 8G 0 part / -nvme0n1p128 259:3 0 1M 0 partThe following is example output for a T2 instance. The root device is /dev/xvda. The attached volume is /dev/xvdf, which is not yet mounted.[ec2-user ~]$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT xvda 202:0 0 8G 0 disk -xvda1 202:1 0 8G 0 part / xvdf 202:80 0 10G 0 disk
  3. Determine whether there is a file system on the volume. New volumes are raw block devices, and you must create a file system on them before you can mount and use them. Volumes that were created from snapshots likely have a file system on them already; if you create a new file system on top of an existing file system, the operation overwrites your data.Use one or both of the following methods to determine whether there is a file system on the volume:
    • Use the file -s command to get information about a specific device, such as its file system type. If the output shows simply data, as in the following example output, there is no file system on the device[ec2-user ~]$ sudo file -s /dev/xvdf /dev/xvdf: dataIf the device has a file system, the command shows information about the file system type. For example, the following output shows a root device with the XFS file system.[ec2-user ~]$ sudo file -s /dev/xvda1 /dev/xvda1: SGI XFS filesystem data (blksz 4096, inosz 512, v2 dirs)
    • Use the lsblk -f command to get information about all of the devices attached to the instance.[ec2-user ~]$ sudo lsblk -fFor example, the following output shows that there are three devices attached to the instances—nvme1n1nvme0n1, and nvme2n1. The first column lists the devices and their partitions. The FSTYPE column shows the file system type for each device. If the column is empty for a specific device, it means that the device does not have a file system. In this case, nvme1n1nvme0n1 are both formatted using the XFS file system, while nvme2n1 does not have a file system.NAME FSTYPE LABEL UUID MOUNTPOINT nvme1n1 xfs 7f939f28-6dcc-4315-8c42-6806080b94dd nvme0n1 ├─nvme0n1p1 xfs / 90e29211-2de8-4967-b0fb-16f51a6e464c / └─nvme0n1p128 nvme2n1
    If the output from these commands show that there is no file system on the device, you must create one.
  4. (Conditional) If you discovered that there is a file system on the device in the previous step, skip this step. If you have an empty volume, use the mkfs -t command to create a file system on the volume.WarningDo not use this command if you’re mounting a volume that already has data on it (for example, a volume that was created from a snapshot). Otherwise, you’ll format the volume and delete the existing data.[ec2-user ~]$ sudo mkfs -t xfs /dev/xvdfIf you get an error that mkfs.xfs is not found, use the following command to install the XFS tools and then repeat the previous command:[ec2-user ~]$ sudo yum install xfsprogs
  5. Use the mkdir command to create a mount point directory for the volume. The mount point is where the volume is located in the file system tree and where you read and write files to after you mount the volume. The following example creates a directory named /data.[ec2-user ~]$ sudo mkdir /data
  6. Use the following command to mount the volume at the directory you created in the previous step.[ec2-user ~]$ sudo mount /dev/xvdf /data
  7. Review the file permissions of your new volume mount to make sure that your users and applications can write to the volume. For more information about file permissions, see File security at The Linux Documentation Project.
  8. The mount point is not automatically preserved after rebooting your instance. To automatically mount this EBS volume after reboot, see Automatically mount an attached volume after reboot.

Automatically mount an attached volume after reboot

To mount an attached EBS volume on every system reboot, add an entry for the device to the /etc/fstab file.

You can use the device name, such as /dev/xvdf, in /etc/fstab, but we recommend using the device’s 128-bit universally unique identifier (UUID) instead. Device names can change, but the UUID persists throughout the life of the partition. By using the UUID, you reduce the chances that the system becomes unbootable after a hardware reconfiguration. For more information, see Identify the EBS device.

To mount an attached volume automatically after reboot

  1. (Optional) Create a backup of your /etc/fstab file that you can use if you accidentally destroy or delete this file while editing it.[ec2-user ~]$ sudo cp /etc/fstab /etc/fstab.orig
  2. Use the blkid command to find the UUID of the device.[ec2-user ~]$ sudo blkid /dev/xvda1: LABEL="/" UUID="ca774df7-756d-4261-a3f1-76038323e572" TYPE="xfs" PARTLABEL="Linux" PARTUUID="02dcd367-e87c-4f2e-9a72-a3cf8f299c10" /dev/xvdf: UUID="aebf131c-6957-451e-8d34-ec978d9581ae" TYPE="xfs"For Ubuntu 18.04 use the lsblk command.[ec2-user ~]$ sudo lsblk -o +UUID
  3. Open the /etc/fstab file using any text editor, such as nano or vim.[ec2-user ~]$ sudo vim /etc/fstab
  4. Add the following entry to /etc/fstab to mount the device at the specified mount point. The fields are the UUID value returned by blkid (or lsblk for Ubuntu 18.04), the mount point, the file system, and the recommended file system mount options. For more information, see the manual page for fstab (run man fstab).UUID=aebf131c-6957-451e-8d34-ec978d9581ae /data xfs defaults,nofail 0 2NoteIf you ever boot your instance without this volume attached (for example, after moving the volume to another instance), the nofail mount option enables the instance to boot even if there are errors mounting the volume. Debian derivatives, including Ubuntu versions earlier than 16.04, must also add the nobootwait mount option.
  5. To verify that your entry works, run the following commands to unmount the device and then mount all file systems in /etc/fstab. If there are no errors, the /etc/fstab file is OK and your file system will mount automatically after it is rebooted.[ec2-user ~]$ sudo umount /data [ec2-user ~]$ sudo mount -aIf you receive an error message, address the errors in the file.WarningErrors in the /etc/fstab file can render a system unbootable. Do not shut down a system that has errors in the /etc/fstab file.If you are unsure how to correct errors in /etc/fstab and you created a backup file in the first step of this procedure, you can restore from your backup file using the following command.[ec2-user ~]$ sudo mv /etc/fstab.orig /etc/fstab
Categories
AWS REDIS Unix

How to install redis on AWS unix ec2 instance ?

 $ vi /etc/yum.repos.d/epel.repo

 [epel]
 name=Extra Packages for Enterprise Linux 6 - $basearch
 baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
 mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
 failovermethod=priority
 enabled=1
 gpgcheck=0
 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

Step 2: Clean all yum repo and update repolist:

$ yum clean all
$ yum repolist

Step 3: Install Redis on the server by running command

$ yum install redis 

Step 4: Check redis status

service redis status

service redis start
service redis stop
service redis restart

Categories
AWS Operating System Unix

Incredible tutorial to more ‘RAM’ to your AWS EC2 instance

Source : https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-7

 

 

Check the System for Swap Information

Before we begin, we should take a look at our server’s storage to see if we already have some swap space available. While we can have multiple swap files or swap partitions, one should generally be enough.

We can see if the system has any configured swap by using swapon, a general-purpose swap utility. With the -s flag, swapon will display a summary of swap usage and availability on our storage device:

swapon -s

If nothing is returned by the command, then the summary was empty and no swap file exists.

Another way of checking for swap space is with the free utility, which shows us the system’s overall memory usage. We can see our current memory and swap usage (in megabytes) by typing:

free -m
             total       used       free     shared    buffers     cached
Mem:          3953        315       3637          8         11        107
-/+ buffers/cache:        196       3756
Swap:            0          0       4095

As you can see, our total swap space in the system is 0. This matches what we saw with swapon.

Check Available Storage Space

The typical way of allocating space for swap is to use a separate partition that is dedicated to the task. However, altering the partition scheme is not always possible due to hardware or software constraints. Fortunately, we can just as easily create a swap file that resides on an existing partition.

Before we do this, we should be aware of our current drive usage. We can get this information by typing:

df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        59G  1.5G   55G   3% /
devtmpfs        2.0G     0  2.0G   0% /dev
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           2.0G  8.3M  2.0G   1% /run
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup

Note: the -h flag simply tells dh to output drive information in a human-friendly reading format. For example, instead of outputting the raw number of memory blocks in a partition, df -h will tell us the space usage and availability in M (for megabytes) or G (for gigabytes).

As you can see on the first line, our storage partition has 59 gigabytes available, so we have quite a bit of space to work with. Keep in mind that this is on a fresh, medium-sized VPS instance, so your actual usage might be very different.

Although there are many opinions about the appropriate size of a swap space, it really depends on your application requirements and your personal preferences. Generally, an amount equal to or double the amount of memory on your system is a good starting point.

Since my system has 4 gigabytes of memory, and doubling that would take a larger chunk from my storage space than I am willing to part with, I will create a swap space of 4 gigabytes to match my system’s memory.

Create a Swap File

Now that we know our available storage space, we can go about creating a swap file within our filesystem. We will create a file called swapfile in our root (/) directory, though you can name the file something else if you prefer. The file must allocate the amount of space that we want for our swap file.

The fastest and easiest way to create a swap file is by using fallocate. This command creates a file of a preallocated size instantly. We can create a 4 gigabyte file by typing:

sudo fallocate -l 4G /swapfile

After entering your password to authorize sudo privileges, the swap file will be created almost instantly, and the prompt will be returned to you. We can verify that the correct amount of space was reserved for swap by using ls:

ls -lh /swapfile
-rw-r--r-- 1 root root 4.0G Oct 30 11:00 /swapfile

As you can see, our swap file was created with the correct amount of space set aside.

Enable a Swap File

Right now, our file is created, but our system does not know that this is supposed to be used for swap. We need to tell our system to format this file as swap and then enable it.

Before we do that, we should adjust the permissions on our swap file so that it isn’t readable by anyone besides the root account. Allowing other users to read or write to this file would be a huge security risk. We can lock down the permissions with chmod:

sudo chmod 600 /swapfile

This will restrict both read and write permissions to the root account only. We can verify that the swap file has the correct permissions by using ls -lh again:

ls -lh /swapfile
-rw------- 1 root root 4.0G Oct 30 11:00 /swapfile

Now that our swap file is more secure, we can tell our system to set up the swap space for use by typing:

sudo mkswap /swapfile
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=b99230bb-21af-47bc-8c37-de41129c39bf

Our swap file is now ready to be used as a swap space. We can begin using it by typing:

sudo swapon /swapfile

To verify that the procedure was successful, we can check whether our system reports swap space now:

swapon -s
Filename                Type        Size    Used    Priority
/swapfile               file        4194300 0     -1

This output confirms that we have a new swap file. We can use the free utility again to corroborate our findings:

free -m
             total       used       free     shared    buffers     cached
Mem:          3953        315       3637          8         11        107
-/+ buffers/cache:        196       3756
Swap:         4095          0       4095

Our swap has been set up successfully, and our operating system will begin to use it as needed.

Make the Swap File Permanent

Our swap file is enabled at the moment, but when we reboot, the server will not automatically enable the file for use. We can change that by modifying the fstab file, which is a table that manages filesystems and partitions.

Edit the file with sudo privileges in your text editor:

sudo nano /etc/fstab

At the bottom of the file, you need to add a line that will tell the operating system to automatically use the swap file that you created:

/swapfile   swap    swap    sw  0   0

When you are finished adding the line, you can save and close the file. The server will check this file on each bootup, so the swap file will be ready for use from now on.

Categories
AWS AWS SQS

AWS – SQS – LAMBDA – CloudWatch – Slack – DLQ – SNS

I am currently working on a transactional platform where orders are pushed to a Queue SQS then pulled and processed by lambda.

In a problematic situation , lambda triggers AWS SNS which will notify sysOp via email and a Slack Channel. Awesome ! And more importantly a DLQ – Dead-Letter Queue will contain workers waiting to be reprocessed and sent to our Technical Customer Service Department to analyze what went wrong

Categories
AWS AWS SQS javascript

SOLUTION – POST API CALL in AWS SQS-LAMBDA – NODEJS – Javascript

If you have been struggling in amazon aws with the combo Lambda-SQS NodeJS fixing following type of errors :

read ECONNRESET at TLSWrap.onStreamRead

Error: read ECONNRESET at TLSWrap.onStreamRead

 

ERROR Uncaught Exception {“errorType”:”Error”,”errorMessage”:”getaddrinfo ENOTFOUND https://xxxx.xxxxxr.com https://xxxxx.xxxxx.com:443″,”code”:”ENOTFOUND”,”stack”:[“Error: getaddrinfo ENOTFOUND

 

ERROR Invoke Error {“errorType”:”TypeError [ERR_INVALID_ARG_TYPE]”,”errorMessage”:”The first argument must be one

Here is the solution

exports.handler = async (event) => {
     try {
         const res = await axios.post('https://davidraleche.com/v1/test',JSON.stringify({}), {
         headers: {
             'Content-Type': 'application/json'
         }})
         console.log(res)
         return {
             statusCode: 200,
             body: res.data
         }
     } catch (e) {
         console.log(e)
         return {
             statusCode: 400,
             body: JSON.stringify(e)
         }
     }
};

 

Categories
AWS AWS SQS BEST PHP CODE

AWS SQS Library

Image result for aws

Interface Contract

<?php

namespace David\App\Models;

interface Queue
{
    public function push(array $message);
    public function pop() : array;
    public function flush(string $queueUrl);
    public function process();
}

 

QueueService Class

<?php

namespace App\Core\Services;

use App\Core\Prelude;
use Aws\Sqs\SqsClient;
use Mapi\App\Models\Queue;

/**
 * Class QueueService
 *
 * @package App\Core\Services
 */
class QueueService extends \App\Core\Classes\Service implements Queue
{
    /**
     * QueueConfigs
     *
     * @var bool
     */
    private $queueConfigs = false;
    /**
     * DbConnections
     *
     * @var bool
     */
    private $dbConnections = false;
    /**
     * SqsClient
     *
     * @var SqsClient|bool
     */
    private $sqsClient  = false;

    /**
     * GetSqsClient
     *
     * @return SqsClient|bool|static
     */
    public function getSqsClient()
    {
        return $this->sqsClient;
    }

    /**
     * SetSqsClient
     *
     * @param SqsClient|bool|static $sqsClient
     */
    public function setSqsClient($sqsClient)
    {
        $this->sqsClient = $sqsClient;
    }

    /**
     * StorageService constructor.
     */
    public function __construct($queue_name, Prelude $prelude)
    {
        if ($queue_name === null) {
            die(
                'QueueService::constructor queue_name does not exist');
        }
        $this->prelude = $prelude;
        $this->queues = $this->queueConfigs()['queues'];
        $this->selectedQueue =  $this->queues[$queue_name];
        $this->queueUrl = $this->selectedQueue['queue_url'] ;
        $this->sqsCredentials = array(
            'region' => $this->selectedQueue['region'],
            'version' => $this->selectedQueue['version'],
            'credentials' => array(
                'key' => $this->selectedQueue['key'],
                'secret' =>$this->selectedQueue['secret'],
            ),
        );

        $this->sqsClient = $this->queueConnect($queue_name);
    }

    /**
     * Storage Configs
     *
     * @return bool|mixed
     */
    private function queueConfigs()
    {
        if ($this->queueConfigs === false) {
            $this->queueConfigs = yaml_parse_file(__DIR__.'/../../config/storage.yaml');
        }

        return $this->queueConfigs;
    }

    /**
     * Queue connection
     *
     * @param string $schema
     * @param bool $allowShared
     */
    public function queueConnect()
    {
        try {
            // Instantiate the client
            $sqsClient = SqsClient::factory($this->sqsCredentials);
        } catch (Exception $e) {
            die('Error Connecting to the Queue '.$e->getMessage());
        }

        return $sqsClient;
    }

    /**
     * Push
     *
     * @param string $jsonMessage
     *
     * @return bool
     */
    public function push(array $params) : bool
    {
        try {
            // Send the message
            $this->sqsClient->sendMessage(
                $params
            );
        } catch (Exception $e) {
            die('Error sending message to queue '.$e->getMessage());
            return false;
        }
        return true;
    }

    public function deleteMessage($result)
    {
        $this->sqsClient->deleteMessage(
            [
                'QueueUrl' => $this->queueUrl, // REQUIRED
                'ReceiptHandle' => $result->get('Messages')[0]['ReceiptHandle'] // REQUIRED
            ]
        );
    }

    /**
     * Pop Message
     *
     * @return array
     */
    public function pop() : array
    {
        $resultMessage = array();
        try {
            $result = $this->sqsClient->receiveMessage(
                array(
                    'AttributeNames' => ['SentTimestamp'],
                    'MaxNumberOfMessages' => 1,
                    'MessageAttributeNames' => ['All'],
                    'QueueUrl' => $this->queueUrl,
                    'WaitTimeSeconds' => 0,
                )
            );
            //Logging Pending Queue Messages
            $this->prelude->log->debug('QueueService:pop COUNT MESSAGES '
                    .count($result->getPath('Messages')));

            if (is_array($result->get('Messages'))) {
                if (count($result->get('Messages')) > 0) {
//                    $this->prelude->log->debug('QueueService:pop'
//                        .json_encode($result->get('Messages')));

                    $status = true;
                    $resultMessage = $result->get('Messages')[0]['Body'];
                    //Delete Message
                    $this->deleteMessage($result);
                }
            } else {
                $status = false;
                $resultMessage = "No messages in queue. \n";
            }
        } catch (Exception $e) {
            die(' message to queue '.$e->getMessage());
        }
        // Add Debugging SQS QUEUE
        $this->prelude->log->debug('WORKER BODY MESSAGE '.$resultMessage);
        return array('status' => $status, 'result' => $resultMessage);
    }