iSIZE BitSave Container on Amazon Marketplace

iSize Uncategorized

Set up iSIZE BitSave Container on Amazon Marketplace

This is a step-by-step guide to set up the iSIZE BitSave Container for Linux running on Amazon Elastic Container (ECS). In the first part of this guide, we will show you how to set up a single instance cluster and setup IAM policy to run tasks in that container. If you already have experience setting us cluster and running tasks, please jump to the second part of the guide.

AWS ECS Cluster

Before you even setup your cluster, you need to create an IAM role that the cluster will run as. The iSIZE tasks we schedule later will also require these permissions to operate properly. Fortunately, there is an easy way to automate this via CloudFormation. Click the CloudFormation template provided in our marketplace listing or simply click here. This will open a stack creation form.

Simply put “isize” as the stack name and agree to the creation of IAM resources and then click the “Create stack” button. This will initiate the IAM creation process and will say “CREATE_IN_PROGRESS” on the left panel. You can refresh and see current progress by clicking the refresh button on the top right.

The process will take about a minute to complete, so refresh and when it says “CREATE_COMPLETE” on the left, click the “Output” tab. Here you can see the created IAM role that we will be using. Copy the part on the right of “/” as shown in the screenshot. You will need it later.

You can now setup your Elastic Container Service (ECS) cluster from the AWS console. We can open the cluster console in another window or tab so we can come back to the created IAM role output later. Make sure you are in the right region where you want to create your cluster by selecting the correct region from the top right of your console. Select “EC2 Linux + Networking” and click the blue “Next step” button at the bottom.

You will have to specify your cluster properties here. Use cluster name: “isize-bitsave”, model: “On-Demand Instance”, EC2 instance type: “g4dn.8xlarge”, EBS Volume Size: “30”, Key pair to a previously create key or create a new one, Security Group port range: “22-8001”.  Then set Container Instance IAM Role to the previous role created via the ClouldFormation template. Finally, click “Create”. We have used the recommended GPU instance type “g4dn.8xlarge” here. The full list of recommended instance types can be found below here.

This will initiate the cluster creation process. This may take a few minutes to complete. When the cluster is created, all three steps will turn green. You are now able to view the cluster by clicking the blue “View Cluster” button.

The cluster status should show a green “ACTIVE”. Click the tab “ECS Instances” to see if a new instance has started for this cluster.

This will show the number of instances running in this cluster. Sometimes when you create a cluster, this tab may be empty. You need to wait for a few minutes until the instance starts and is fully initiated before you can assign tasks (job container) to it. Refresh the list periodically until the instance list is populated with an “ACTIVE” status instance.

Create a new task definition that will run on ECS via EC2. Click “Next step” and go to the bottom of the next page.

You should see a button here with the label “Configure via JSON”. Copy the GPU task JSON from the bottom of this guide here and paste it in the popup after clicking the configure button.

Replace the text <CONTAINER_LOCATION> with the actual location of the container provided to you when you subscribed to iSIZE BitSave container and click save. This should have populated all the fields required to pull the iSIZE container, set up sharing, environment etc. The only thing you need to change here is to select the IAM role you created before in the two files shows below.

Then go to the bottom and click the blue “Create” button. If everything went properly, you should see a “Created task definition successfully” message. Now it’s time to run this task. Click the dropdown “Actions” and select “Run Task”.

Choose “EC2” as the Launch type and click the blue “Run Task” button at the bottom.

You should see a “Created tasks successfully” message as well as a Tasks tab in the bottom showing the task as pending. Setting up a task takes about 10 minutes, so you need to wait for a bit.

Refresh the task list every few minutes until you see the status as green “RUNNING”. At this point the iSIZE container is ready to accept jobs. To find out the public DNS address of your instance click the “Container instance” ID.

The “Public DNS” address is where the iSIZE BitSave container will accept REST commands. To do a quick test, open a terminal or command prompt and enter the command:

curl -X GET http://PUBLIC_DNS:8000/isize

If you get an empty array response “[]” then the command was successful.

Important considerations

  • The cluster instance folder “/home/ec2-user” is shared with the container as “/isize_files/”
  • The container reads data from the instance volume. Set EBS Volume size to a higher value if you intend to process large files.
  • You need to SFTP to the instance to transfer video files there before the container is able to access them. Another solution is to create an EFS Volume with all your media and automount it with your container task. This requires the creation of the EFS volume, proper IAM policy, security group and significant changes to the task configuration JSON on your part.
  • We have allowed port range 22-8001 to simplify the creation of the ECS cluster. In a production environment, you should secure your cluster by modifying the security group to allow only three ports, port 22 (for SSH), port 8000 (for the iSIZE REST API) and port 8001 (for receiving live video streams).

REST API Description

The iSIZE BitSave REST API provides an interface and functionality to integrate precoding within your existing system. The API allows the creation of precoding jobs as well as poll status of jobs using standard REST clients.

By default, the iSIZE BitSave Container listens to port 8000 for REST commands. The REST API does not require any authentication and accepts commands via

[container_ip_addess]:[port]/[function]/[parameter]

The API is used for all the functions available below:

FunctionDescription
POST /isizeAdd a new job to queue for processing
GET /isizeGet status of all job in the queue
GET /isize/{job_ID}Get the detailed status of a single job in the queue specified by job_ID
DELETE /isize/{job_ID}Delete a job specified by job_ID
POST /codecEnable additional optional codecs
GET /versionGet version information of iSIZE BitSave Container

REST API function ‘add job’

This function adds precoding jobs to the queue. The JSON job details must be submitted via: POST /isize The JSON job format used is:

{ 
"input": "/isize_files/test.mp4", 
"output": "/isize_files/output/test_output.mp4",
"isize_precoding": "PRECODER_ADAPTIVE",
"isize_use_gpu": "TRUE",
"isize_mode": "VoD",
"codec": "AV1",
"bitrate": "4000",
"width": "1920",
"height": "1080",
"gop": "150",
"pix_fmt": "yuv420p",
"preset": "slow",
"prores_codec_profile": "PRORES_422_PROXY"
}

The details of each of the parameters are below:

ParameterRequiredDescription
inputRequiredFull path to input file available directly to container
outputRequiredOutput file path directly writable to container
isize_precodingRequiredPrecoder model to use. Accepts: PRECODER_LOW, PRECODER_BAL, PRECODER_ADAPTIVE
isize_use_gpuOptionalAvailable only on a GPU instance, otherwise queued jobs will fail. Accepts: TRUE, FALSE. Default: FALSE
isize_modeOptionalSets the processing mode, video on demand or Live streaming. Accepts: VoD, Live. Default: VoD
codecRequiredThe codec to use. Accepts: Y4M, VP9, AV1, PRORES. With additional codec activation (see below) also accepts: H_264, H_264_M4V, MPEG2VIDEO, H_265
bitrateRequiredEncoding bitrate in kbps. Example: 5000 is 5Mbps. Ignored for codec=PRORES
widthOptionalAccepts ‘source’ or valid numerical width
heightOptionalAccepts ‘source’ or valid numerical height
gopRequiredKeyfram interval. Accepts: 1-10000
pix_fmtOptionalOutput pixel format. Accepts: yuv420p or yuv420p10le. Ignored for codec=PRORES
presetOptionalEncoder preset. Accepts: “medium”, “slow”, “slower” and “veryslow” for H_264, H_265 or “1”-“8” for AV1
prores_codec_profileOptionalOnly required when codec=PRORES. Accepts: PRORES_422_PROXY, PRORES_422_LT, PRORES_422, PRORES_422_HQ, PRORES_444, PRORES_444_XQ

Returns:

HTTP return codeResponseDescription
201{“input”:”input_file”, “job_id”: created_id, “output”:”output_file”}Job successfully submitted and queued for processing
400{“input”:”input_file”, “error”: “Text description of error”}There was an error queuing the job. Please check the “error” text on how to fix this

 

REST API function ‘list jobs’

Get the list of jobs from the queue. GET /isize Returns:

HTTP return codeResponseDescription
200{“job_id”:”job_id”, “input”:”input_file”, “isize_precoding”:”isize_precoding_model_used”, “codec”:”codec_used”, “status_text”:”status_of_job”, “progress”:”progress_counter”, “start_time”:”start_time YYYY-MM-DD HH:mm:ss”, “update_time”:”update_time YYYY-MM-DD HH:mm:ss”, “end_time”:”end_time YYYY-MM-DD HH:mm:ss”}Returns blank if job queue is empty or an array of jobs. This retuns queued jobs, active jobs as well as completed jobs.

REST API function ‘list job’

Get the details of a single job from the queue. GET /isize/job_id Returns:

HTTP return codeResponseDescription
200{“job_id”:”job_id”, “input”:”input_file”, “output”:$encode_dst_url, “isize_precoding”:”isize_precoding_model_used”, “isize_use_gpu”:$encode_use_gpu, “codec”:”codec_used”, “bitrate”:”encode_bitrate_used”, “width”:”output_width”, “height”:”output_height”, “gop”:”keyframe_interval”, “pix_fmt”:”pixel_format_used”, “prores_codec_profile”:”prores_profile_used”, “status_text”:”status_of_job”, “progress”:”progress_counter”, “start_time”:”start_time YYYY-MM-DD HH:mm:ss”, “update_time”:”update_time YYYY-MM-DD HH:mm:ss”, “end_time”:”end_time YYYY-MM-DD HH:mm:ss”}Returns complete details about the job.
404{“error”:”Invalid Job ID”}The specified job ID is not valid

 

REST API function ‘delete job’

Delete an already completed job or a queued job. You can’t delete in-progress jobs. DELETE /isize/job_id Returns:

HTTP return codeResponseDescription
200{“success”:”Job Deleted”}Job deleted successfully
404{“error”:”Invalid Job ID”}The specified job ID is not valid

 

REST API function ‘enable additional codecs’

Out of the box, the iSIZE BitSave supports Y4M, VP9, AV1 and PRORES codec. You can sent POST request to activate optional additional components (H_264, H_264_M4V, MPEG2VIDEO, H_265). By running the POST activate_encoders command, the below list of optional software components will be installed and activated in the container. Please note that, by running this activation command, you are accepting their corresponding licenses: x264 source & license: https://code.videolan.org/videolan/x264/ x264 config: –enable-shared –enable-pic –bit-depth=all –chroma-format=all x265 source & license: https://github.com/videolan/x265 x265 config: default (from x265/build/linux/multilib.sh) FFmpeg souce & license: https://github.com/FFmpeg/FFmpeg FFmpeg config: –disable-stripping –enable-swresample –enable-libopus –enable-libvorbis –enable-libvpx –enable-libvmaf –enable-gnutls –enable-libaom –enable-libsvtav1 –pkg-config-flags=”–static” –enable-shared –enable-gpl –enable-libx264 –enable-libx265.

The POST request to activate these optional additional components is:

POST /codec

The JSON message format used is:

{ 
"enable_codecs":"TRUE" 
}

Returns:

HTTP return codeResponseDescription
200{“success”:”Codec enabling in place. This may take up to 10 minutes. Please wait 10 minutes before sending additional requests.”}Optional additional codec support is being activated. Please wait 10 minutes before sending any new job requests. You only need to enable optional codecs once per running container.
400{“errir”:”Invalid request”}The request was not sent properly. Please recheck if the submitted JSON value enable_codec is TRUE.

Sample REST commands

You can use CURL to send sample REST commands to iSIZE BitSave API.

curl -v -H "Content-Type: application/json" -X POST --data @isize_job.json "http://<CONTAINER_INSTANCE_IP>:8000/isize"

The content of isize_job.json should contain the job specifics. Some sample job parameters are shown below:

Take the sample input file “/isize/samples/test.mp4” and apply iSIZE “PRECODER_BAL” precoding, resize to 1280×720 and produce output in the shared folder as lossless Y4M file “/isize_files/test_output_resized.y4m” using GPU.

{ 
    "input": "/isize/samples/test.mp4",		
    "output": "/isize_files/test_output_resized.y4m",
    "isize_precoding": "PRECODER_BAL",
    "isize_use_gpu": "TRUE",
    "codec": "Y4M",
    "width": "1280",
    "height": "720",
    "pix_fmt": "yuv420p10le"
 }

Take the sample input file “/isize/samples/test.mp4” and apply iSIZE “PRECODER_LOW” precoding, encode it as AV1 with 2Mbps bitrate and produce output in the shared folder as webm file “/isize_files/test_output_av1.webm” using CPU.

{ 
    "input": "/isize/samples/test.mp4",		
    "output": "/isize_files/test_output_av1.webm",
    "isize_precoding": "PRECODER_LOW",
    "isize_use_gpu": "FALSE",
    "codec": "AV1",
    "bitrate": "2000",
    "gop": "60"
}

List current jobs:

curl -X GET http://<CONTAINER_INSTANCE_IP>:8000/isize

Transfer Files

iSIZE container needs direct access to your files before it can process them. You should transfer source media files to the cluster instance before queuing jobs. Similarly, you can get the output files from the instance the same way. You can use any mechanism allowed in AWS to transfer files but SFTP is the most secure and common. A free and popular such SFTP client is FileZilla. Download and install FileZilla and run it. Click “Open Site Manager”.

Click “New Site” and enter a site name. On the right site select and enter the appropriate information. The “<CONTAINER_IP>” needs to be the instance IP in your cluster running the task and “Key file” needs to be the same key that you used when setting up the cluster. Click Connect.


This will connect to the instance. Now you can transfer files to and from the instance+container.




Sample ECS task JSON for GPU instances

The JSON file below can be customized and used to setup and run iSIZE BitSave container in ECS GPU instances. Please make sure you set up appropriate volumes and mount points.

{
    "requiresCompatibilities": [
        "EC2"
    ],
    "inferenceAccelerators": [],
    "containerDefinitions": [
        {
            "name": "isize_bitsave",
            "image": "<CONTAINER_LOCATION>",
            "memoryReservation": "20000",
            "resourceRequirements": [
                {
                    "type": "GPU",
                    "value": "1"
                }
            ],
            "essential": true,
            "portMappings": [
                {
                    "hostPort": "8000",
                    "containerPort": "8000",
                    "protocol": "tcp"
                },
                {
                    "hostPort": "8001",
                    "containerPort": "8001",
                    "protocol": "tcp"
                }
            ],
            "environment": null,
            "environmentFiles": [],
            "secrets": null,
            "mountPoints": [
                {
                    "sourceVolume": "isize_shared_folder",
                    "containerPath": "/isize_files",
                    "readOnly": ""
                }
            ],
            "volumesFrom": null,
            "hostname": null,
            "user": null,
            "workingDirectory": null,
            "extraHosts": null,
            "logConfiguration": null,
            "ulimits": null,
            "dockerLabels": null,
            "dependsOn": null,
            "repositoryCredentials": {
                "credentialsParameter": ""
            }
        }
    ],
    "volumes": [
        {
            "name": "isize_shared_folder",
            "host": {
                "sourcePath": "/home/ec2-user"
            }
        }
    ],
    "networkMode": null,
    "memory": "30Gb",
    "cpu": null,
    "placementConstraints": [],
    "family": "isize-task",
    "tags": [],
    "taskRoleArn": "<IAM_ROLE_CREATED_BEFORE>"
}

Sample ECS task JSON for CPU instances

The JSON file below can be customized and used to setup and run iSIZE BitSave container in ECS CPU instances. Please make sure you set up appropriate volumes and mount points.

{
    "requiresCompatibilities": [
        "EC2"
    ],
    "inferenceAccelerators": [],
    "containerDefinitions": [
        {
            "name": "isize_bitsave",
            "image": "<CONTAINER_LOCATION>",
            "memoryReservation": "20000",
            "resourceRequirements": null,
            "essential": true,
            "portMappings": [
                {
                    "hostPort": "8000",
                    "containerPort": "8000",
                    "protocol": "tcp"
                },
                {
                    "hostPort": "8001",
                    "containerPort": "8001",
                    "protocol": "tcp"
                }
            ],
            "environment": null,
            "environmentFiles": [],
            "secrets": null,
            "mountPoints": [
                {
                    "sourceVolume": "isize_shared_folder",
                    "containerPath": "/isize_files",
                    "readOnly": ""
                }
            ],
            "volumesFrom": null,
            "hostname": null,
            "user": null,
            "workingDirectory": null,
            "extraHosts": null,
            "logConfiguration": null,
            "ulimits": null,
            "dockerLabels": null,
            "dependsOn": null,
            "repositoryCredentials": {
                "credentialsParameter": ""
            }
        }
    ],
    "volumes": [
        {
            "name": "isize_shared_folder",
            "host": {
                "sourcePath": "/home/ec2-user"
            }
        }
    ],
    "networkMode": null,
    "memory": "30Gb",
    "cpu": null,
    "placementConstraints": [],
    "family": "isize-task",
    "tags": [],
    "taskRoleArn": "<IAM_ROLE_CREATED_BEFORE>"
}

Recommended Instances

The following instance types are recommended:

m6g.8xlarge
m6g.12xlarge
m6gd.8xlarge
m6gd.12xlarge
m5.8xlarge
m5.12xlarge
m5d.8xlarge
m5d.12xlarge
c6g.8xlarge
c6g.12xlarge
c5.4xlarge
c5.9xlarge
c5.12xlarge
c5ad.4xlarge
c5ad.8xlarge
c5ad.12xlarge
c5d.4xlarge
c5d.9xlarge  (recommended CPU instance)
c5d.12xlarge
c4.8xlarge
g4dn.4xlarge
g4dn.8xlarge   (recommended GPU instance)
g4dn.12xlarge
g3.8xlarge
g3.16xlarge
r5.8xlarge
r5.12xlarge
r5a.8xlarge
r5a.12xlarge
r5d.8xlarge
r5d.12xlarge
r5dn.4xlarge
r5dn.8xlarge
r4.8xlarge
r4.16xlarge

For optimal performance, we recommend “g4dn.8xlarge” instance for GPU mode and “c5d.9xlarge” instance for CPU mode.

Technical Support

If you need additional assistance please contact support@isize.co