Set up iSIZE BitSave Container on Amazon Marketplace
This is a step-by-step guide to set up the iSIZE BitSave Container for Linux running on Amazon Elastic Container (ECS). In the first part of this guide, we will show you how to set up a single instance cluster and setup IAM policy to run tasks in that container. If you already have experience setting us cluster and running tasks, please jump to the second part of the guide.
Requirements
- ECS Cluster. iSIZE BitSave Container is supported in all regions where ECS cluster is supported.
- Minimum of one instance running under the cluster
- IAM role with proper permission
- “AmazonEC2ContainerServiceforEC2Role” for running EC2 container
- “AWSMarketplaceMeteringFullAccess” for billing/licensing
- “AWSMarketplaceMeteringRegisterUsage” for registering the task for billing.
The above requirements will be fulfilled automatically using the CloudFormation template provided below or you can also set them up manually. The expected time to complete the whole setup is 10 minutes.
Architecture Diagram
The following diagram shows the typical customer deployment with the relationship between iSIZE Bitsave, AWS services, and networks/subnets used.
AWS ECS Cluster
Before you even setup your cluster, you need to create an IAM role that the cluster will run as. The iSIZE tasks we schedule later will also require these permissions to operate properly. Fortunately, there is an easy way to automate this via CloudFormation. Click the CloudFormation template provided in our marketplace listing or simply click here. This will open a stack creation form.
Simply put “isize” as the stack name and agree to the creation of IAM resources and then click the “Create stack” button. This will initiate the IAM creation process and will say “CREATE_IN_PROGRESS” on the left panel. You can refresh and see current progress by clicking the refresh button on the top right.
The process will take about a minute to complete, so refresh and when it says “CREATE_COMPLETE” on the left, click the “Output” tab. Here you can see the created IAM role that we will be using. Copy the part on the right of “/” as shown in the screenshot. You will need it later.
You can now setup your Elastic Container Service (ECS) cluster from the AWS console. We can open the cluster console in another window or tab so we can come back to the created IAM role output later. Make sure you are in the right region where you want to create your cluster by selecting the correct region from the top right of your console. Select “EC2 Linux + Networking” and click the blue “Next step” button at the bottom.
You will have to specify your cluster properties here. Use cluster name: “isize-bitsave”, model: “On-Demand Instance”, EC2 instance type: “g4dn.8xlarge”, EBS Volume Size: “30”, Key pair to a previously create key or create a new one, Security Group port range: “22-8001”. Then set Container Instance IAM Role to the previous role created via the ClouldFormation template. Finally, click “Create”. We have used the recommended GPU instance type “g4dn.8xlarge” here. The full list of recommended instance types can be found below here. Note that you will need a very large EBS Volume size if you wish to use the container to send jobs directly to MediaConvert.
This will initiate the cluster creation process. This may take a few minutes to complete. When the cluster is created, all three steps will turn green. You are now able to view the cluster by clicking the blue “View Cluster” button.
The cluster status should show a green “ACTIVE”. Click the tab “ECS Instances” to see if a new instance has started for this cluster.
This will show the number of instances running in this cluster. Sometimes when you create a cluster, this tab may be empty. You need to wait for a few minutes until the instance starts and is fully initiated before you can assign tasks (job container) to it. Refresh the list periodically until the instance list is populated with an “ACTIVE” status instance.
Create a new task definition that will run on ECS via EC2. Click “Next step” and go to the bottom of the next page.
You should see a button here with the label “Configure via JSON”. Copy the GPU task JSON from the bottom of this guide here and paste it in the popup after clicking the configure button.
Replace the text <CONTAINER_LOCATION> with the actual location of the container provided to you when you subscribed to iSIZE BitSave container and click save. This should have populated all the fields required to pull the iSIZE container, set up sharing, environment etc. The only thing you need to change here is to select the IAM role you created before in the two files shows below.
Then go to the bottom and click the blue “Create” button. If everything went properly, you should see a “Created task definition successfully” message. Now it’s time to run this task. Click the dropdown “Actions” and select “Run Task”.
Choose “EC2” as the Launch type and click the blue “Run Task” button at the bottom.
You should see a “Created tasks successfully” message as well as a Tasks tab in the bottom showing the task as pending. Setting up a task takes about 10 minutes, so you need to wait for a bit.
Refresh the task list every few minutes until you see the status as green “RUNNING”. At this point the iSIZE container is ready to accept jobs. To find out the public DNS address of your instance click the “Container instance” ID.
The “Public DNS” address is where the iSIZE BitSave container will accept REST commands. To do a quick test, open a terminal or command prompt and enter the command:
curl -X GET http://PUBLIC_DNS:8000/isize
If you get an empty array response “[]” then the command was successful.
Important considerations
- The cluster instance folder “/home/ec2-user” is shared with the container as “/isize_files/”. This is where all your data is stored.
- The container reads data from the instance volume. Set EBS Volume size to a higher value if you intend to process large files.
- You need to SFTP to the instance to transfer video files there before the container is able to access them. Another solution is to create an EFS Volume with all your media and automount it with your container task. This requires the creation of the EFS volume, proper IAM policy, security group and significant changes to the task configuration JSON on your part.
- We have allowed port range 22-8001 to simplify the creation of the ECS cluster. In a production environment, you should secure your cluster by modifying the security group to allow only three ports, port 22 (for SSH), port 8000 (for the iSIZE REST API) and port 8001 (for receiving live video streams).
- The total process will create one new IAM role, one new ECS cluster, one new security group and a minimum of one EC2 instance with an elastic IP attached. Check your AWS service quotas to make sure you have enough limit to do so.
Billing
As soon as your ECS Task is running, you will be billed based on the number of ECS tasks running. It does not matter how many videos you precode, the bill is based on the amount of time the ECS task runs. The current pricing information can be found here.
REST API Description
The iSIZE BitSave REST API provides an interface and functionality to integrate precoding within your existing system. The API allows the creation of precoding jobs as well as poll status of jobs using standard REST clients.
By default, the iSIZE BitSave Container listens to port 8000 for REST commands. The REST API does not require any authentication and accepts commands via
[container_ip_addess]:[port]/[function]/[parameter]
The API is used for all the functions available below:
| Function | Description |
| POST /isize | Add a new job to queue for processing |
| POST /isize_mc | Add a new job to queue for processing and then passing to MediaConvert |
| GET /isize | Get status of all job in the queue |
| GET /isize/{job_ID} | Get the detailed status of a single job in the queue specified by job_ID |
| DELETE /isize/{job_ID} | Delete a job specified by job_ID |
| POST /codec | Enable additional optional codecs |
| GET /version | Get version information of iSIZE BitSave Container |
REST API function ‘add job’
This function adds precoding jobs to the queue. The JSON job details must be submitted via: POST /isize The JSON job format used is:
{
"input": "/isize_files/test.mp4",
"output": "/isize_files/output/test_output.mp4",
"isize_precoding": "PRECODER_ADAPTIVE",
"isize_use_gpu": "TRUE",
"isize_mode": "VoD",
"codec": "AV1",
"bitrate": "4000",
"width": "1920",
"height": "1080",
"gop": "150",
"pix_fmt": "yuv420p",
"preset": "slow",
"prores_codec_profile": "PRORES_422_PROXY"
}
The details of each of the parameters are below:
| Parameter | Required | Description |
| input | Required | Full path to input file available directly to container |
| output | Required | Output file path directly writable to container |
| isize_precoding | Required | Precoder model to use. Accepts: PRECODER_LOW, PRECODER_BAL, PRECODER_ADAPTIVE |
| isize_use_gpu | Optional | Available only on a GPU instance, otherwise queued jobs will fail. Accepts: TRUE, FALSE. Default: FALSE |
| isize_mode | Optional | Sets the processing mode, video on demand or Live streaming. Accepts: VoD, Live. Default: VoD |
| codec | Required | The codec to use. Accepts: Y4M, VP9, AV1, PRORES. With additional codec activation (see below) also accepts: H_264, H_264_M4V, MPEG2VIDEO, H_265 |
| bitrate | Required | Encoding bitrate in kbps. Example: 5000 is 5Mbps. Ignored for codec=PRORES |
| width | Optional | Accepts ‘source’ or valid numerical width |
| height | Optional | Accepts ‘source’ or valid numerical height |
| gop | Required | Keyfram interval. Accepts: 1-10000 |
| pix_fmt | Optional | Output pixel format. Accepts: yuv420p or yuv420p10le. Ignored for codec=PRORES |
| preset | Optional | Encoder preset. Accepts: “medium”, “slow”, “slower” and “veryslow” for H_264, H_265 or “1”-“8” for AV1 |
| prores_codec_profile | Optional | Only required when codec=PRORES. Accepts: PRORES_422_PROXY, PRORES_422_LT, PRORES_422, PRORES_422_HQ, PRORES_444, PRORES_444_XQ |
Returns:
| HTTP return code | Response | Description |
| 201 | {“input”:”input_file”, “job_id”: created_id, “output”:”output_file”} | Job successfully submitted and queued for processing |
| 400 | {“input”:”input_file”, “error”: “Text description of error”} | There was an error queuing the job. Please check the “error” text on how to fix this |
REST API function ‘add job and send to AWS MediaConvert’
This function adds precoding jobs to the queue and then passes the precoded file to AWS MediaConvert for encoding. The JSON job details must be submitted via: POST /isize_mc The JSON job format is the same as any valid AWS MediaConvert create job JSON with some additional fields required:
{
"aws_accesskey": "<YOUR_AWS_ACCESSKEY>",
"aws_secretkey": "<YOUR_AWS_SECRETKEY>",
"aws_region": "<AWS_MEDIACONVERT_REGION>",
"aws_mediaconvert_endpoint": "<AWS_MEDIACONVERT_REGION>",
}
The details of each of the parameters are below:
| Parameter | Required | Description |
| aws_accesskey | Required |
Your AWS AccessKey. The AccessKey account must have proper permissions to:
|
| aws_secretkey | Required | Your AWS SecretKey |
| aws_region | Required | AWS region of your MediaConvert endpoint |
| aws_mediaconvert_endpoint | Required | Your MediaConvert endpoint in the format: https://<name>.mediaconvert.<region>.amazonaws.com |
Returns:
| HTTP return code | Response | Description |
| 201 | {“input”:”input_file”, “job_id”: created_id, “output”:”output_file”} | Job successfully submitted and queued for processing |
| 400 | {“input”:”input_file”, “error”: “Text description of error”} | There was an error queuing the job. Please check the “error” text on how to fix this |
Example JSON settings to perform iSIZE precoding and then encode to H264 (AVC) format SINGLE PASS at 6Mbps:
{
"aws_accesskey": "<YOUR_AWS_ACCESSKEY>",
"aws_secretkey": "<YOUR_AWS_SECRETKEY>",
"aws_region": "<AWS_MEDIACONVERT_REGION>",
"aws_mediaconvert_endpoint": "<AWS_MEDIACONVERT_ENDPOINT>",
"Queue": "arn:aws:mediaconvert:<QUEUE_ADDRESS>",
"UserMetadata": {},
"Role": "arn:aws:iam::<MEDIACONVERT_IAM_ADDRESS>",
"Settings": {
"TimecodeConfig": {
"Source": "ZEROBASED"
},
"OutputGroups": [
{
"Name": "File Group",
"Outputs": [
{
"ContainerSettings": {
"Container": "MP4",
"Mp4Settings": {
"CslgAtom": "INCLUDE",
"CttsVersion": 0,
"FreeSpaceBox": "EXCLUDE",
"MoovPlacement": "PROGRESSIVE_DOWNLOAD"
}
},
"VideoDescription": {
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 0,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"ParNumerator": 1,
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"GopClosedCadence": 1,
"HrdBufferInitialFillPercentage": 90,
"GopSize": 2,
"Slices": 4,
"GopBReference": "DISABLED",
"HrdBufferSize": 6000000,
"MaxBitrate": 6000000,
"SlowPal": "DISABLED",
"ParDenominator": 1,
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "ENABLED",
"EntropyEncoding": "CABAC",
"FramerateControl": "INITIALIZE_FROM_SOURCE",
"RateControlMode": "QVBR",
"QvbrSettings": {
"QvbrQualityLevel": 7,
"QvbrQualityLevelFineTune": 0
},
"CodecProfile": "HIGH",
"Telecine": "NONE",
"MinIInterval": 0,
"AdaptiveQuantization": "LOW",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS_HQ",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "SECONDS",
"ParControl": "SPECIFIED",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "ADAPTIVE"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"AudioDescriptions": [
{
"AudioTypeControl": "FOLLOW_INPUT",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"AudioDescriptionBroadcasterMix": "NORMAL",
"Bitrate": 96000,
"RateControlMode": "CBR",
"CodecProfile": "LC",
"CodingMode": "CODING_MODE_2_0",
"RawFormat": "NONE",
"SampleRate": 48000,
"Specification": "MPEG4"
}
},
"LanguageCodeControl": "FOLLOW_INPUT"
}
]
}
],
"OutputGroupSettings": {
"Type": "FILE_GROUP_SETTINGS",
"FileGroupSettings": {
"Destination": "s3://<DESTINATION_VIDEO_LOCATION_ADDRESS>"
}
}
}
],
"AdAvailOffset": 0,
"Inputs": [
{
"AudioSelectors": {
"Audio Selector 1": {
"Offset": 0,
"DefaultSelection": "DEFAULT",
"ProgramSelection": 1
}
},
"VideoSelector": {
"ColorSpace": "FOLLOW",
"Rotate": "DEGREE_0",
"AlphaBehavior": "DISCARD"
},
"FilterEnable": "AUTO",
"PsiControl": "USE_PSI",
"FilterStrength": 0,
"DeblockFilter": "DISABLED",
"DenoiseFilter": "DISABLED",
"TimecodeSource": "ZEROBASED",
"FileInput": "s3://<SOURCE_VIDEO_ADDRESS>"
}
]
},
"AccelerationSettings": {
"Mode": "DISABLED"
},
"StatusUpdateInterval": "SECONDS_60",
"Priority": 0
}
Example JSON settings to perform iSIZE precoding and then encode to H265 (HEVC) format MULTIPLE PASS quality level 7 at 5Mbps:
{
"aws_accesskey": "<YOUR_AWS_ACCESSKEY>",
"aws_secretkey": "<YOUR_AWS_SECRETKEY>",
"aws_region": "<AWS_MEDIACONVERT_REGION>",
"aws_mediaconvert_endpoint": "<AWS_MEDIACONVERT_ENDPOINT>",
"Queue": "arn:aws:mediaconvert:<QUEUE_ADDRESS>",
"UserMetadata": {},
"Role": "arn:aws:iam::<MEDIACONVERT_IAM_ADDRESS>",
"Settings": {
"TimecodeConfig": {
"Source": "ZEROBASED"
},
"OutputGroups": [
{
"Name": "File Group",
"Outputs": [
{
"ContainerSettings": {
"Container": "MP4",
"Mp4Settings": {
"CslgAtom": "INCLUDE",
"CttsVersion": 0,
"FreeSpaceBox": "EXCLUDE",
"MoovPlacement": "PROGRESSIVE_DOWNLOAD"
}
},
"VideoDescription": {
"ScalingBehavior": "DEFAULT",
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_265",
"H265Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"GopClosedCadence": 1,
"AlternateTransferFunctionSei": "DISABLED",
"GopSize": 90,
"Slices": 1,
"GopBReference": "DISABLED",
"MaxBitrate": 5000000,
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"FramerateControl": "INITIALIZE_FROM_SOURCE",
"RateControlMode": "QVBR",
"QvbrSettings": {
"QvbrQualityLevel": 7,
"QvbrQualityLevelFineTune": 0
},
"CodecProfile": "MAIN_MAIN",
"Tiles": "ENABLED",
"Telecine": "NONE",
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "MULTI_PASS_HQ",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"TemporalIds": "DISABLED",
"SampleAdaptiveOffsetFilterMode": "ADAPTIVE",
"WriteMp4PackagingType": "HEV1",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"AudioDescriptions": [
{
"AudioTypeControl": "FOLLOW_INPUT",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"AudioDescriptionBroadcasterMix": "NORMAL",
"Bitrate": 96000,
"RateControlMode": "CBR",
"CodecProfile": "LC",
"CodingMode": "CODING_MODE_2_0",
"RawFormat": "NONE",
"SampleRate": 48000,
"Specification": "MPEG4"
}
},
"LanguageCodeControl": "FOLLOW_INPUT"
}
]
}
],
"OutputGroupSettings": {
"Type": "FILE_GROUP_SETTINGS",
"FileGroupSettings": {
"Destination": "s3://<DESTINATION_VIDEO_LOCATION_ADDRESS>"
}
}
}
],
"AdAvailOffset": 0,
"Inputs": [
{
"AudioSelectors": {
"Audio Selector 1": {
"Offset": 0,
"DefaultSelection": "DEFAULT",
"ProgramSelection": 1
}
},
"VideoSelector": {
"ColorSpace": "FOLLOW",
"Rotate": "DEGREE_0",
"AlphaBehavior": "DISCARD"
},
"FilterEnable": "AUTO",
"PsiControl": "USE_PSI",
"FilterStrength": 0,
"DeblockFilter": "DISABLED",
"DenoiseFilter": "DISABLED",
"InputScanType": "AUTO",
"TimecodeSource": "ZEROBASED",
"FileInput": "s3://<SOURCE_VIDEO_ADDRESS>"
}
]
},
"AccelerationSettings": {
"Mode": "DISABLED"
},
"StatusUpdateInterval": "SECONDS_60",
"Priority": 0
}
REST API function ‘list jobs’
Get the list of jobs from the queue. GET /isize Returns:
| HTTP return code | Response | Description |
| 200 | {“job_id”:”job_id”, “input”:”input_file”, “isize_precoding”:”isize_precoding_model_used”, “codec”:”codec_used”, “status_text”:”status_of_job”, “progress”:”progress_counter”, “start_time”:”start_time YYYY-MM-DD HH:mm:ss”, “update_time”:”update_time YYYY-MM-DD HH:mm:ss”, “end_time”:”end_time YYYY-MM-DD HH:mm:ss”} | Returns blank if job queue is empty or an array of jobs. This returns queued jobs, active jobs as well as completed jobs. |
REST API function ‘list job’
Get the details of a single job from the queue. GET /isize/job_id Returns:
| HTTP return code | Response | Description |
| 200 | {“job_id”:”job_id”, “input”:”input_file”, “output”:$encode_dst_url, “isize_precoding”:”isize_precoding_model_used”, “isize_use_gpu”:$encode_use_gpu, “codec”:”codec_used”, “bitrate”:”encode_bitrate_used”, “width”:”output_width”, “height”:”output_height”, “gop”:”keyframe_interval”, “pix_fmt”:”pixel_format_used”, “prores_codec_profile”:”prores_profile_used”, “status_text”:”status_of_job”, “progress”:”progress_counter”, “start_time”:”start_time YYYY-MM-DD HH:mm:ss”, “update_time”:”update_time YYYY-MM-DD HH:mm:ss”, “end_time”:”end_time YYYY-MM-DD HH:mm:ss”} | Returns complete details about the job. |
| 404 | {“error”:”Invalid Job ID”} | The specified job ID is not valid |
REST API function ‘delete job’
Delete an already completed job or a queued job. You can’t delete in-progress jobs. DELETE /isize/job_id Returns:
| HTTP return code | Response | Description |
| 200 | {“success”:”Job Deleted”} | Job deleted successfully |
| 404 | {“error”:”Invalid Job ID”} | The specified job ID is not valid |
REST API function ‘enable additional codecs’
Out of the box, the iSIZE BitSave supports Y4M, VP9, AV1 and PRORES codec. You can sent POST request to activate optional additional components (H_264, H_264_M4V, MPEG2VIDEO, H_265). By running the POST activate_encoders command, the below list of optional software components will be installed and activated in the container. Please note that, by running this activation command, you are accepting their corresponding licenses: x264 source & license: https://code.videolan.org/videolan/x264/ x264 config: –enable-shared –enable-pic –bit-depth=all –chroma-format=all x265 source & license: https://github.com/videolan/x265 x265 config: default (from x265/build/linux/multilib.sh) FFmpeg souce & license: https://github.com/FFmpeg/FFmpeg FFmpeg config: –disable-stripping –enable-swresample –enable-libopus –enable-libvorbis –enable-libvpx –enable-libvmaf –enable-gnutls –enable-libaom –enable-libsvtav1 –pkg-config-flags=”–static” –enable-shared –enable-gpl –enable-libx264 –enable-libx265.
The POST request to activate these optional additional components is:
POST /codec
The JSON message format used is:
{
"enable_codecs":"TRUE"
}
Returns:
| HTTP return code | Response | Description |
| 200 | {“success”:”Codec enabling in place. This may take up to 10 minutes. Please wait 10 minutes before sending additional requests.”} | Optional additional codec support is being activated. Please wait 10 minutes before sending any new job requests. You only need to enable optional codecs once per running container. |
| 400 | {“errir”:”Invalid request”} | The request was not sent properly. Please recheck if the submitted JSON value enable_codec is TRUE. |
Sample REST commands
You can use CURL to send sample REST commands to iSIZE BitSave API.
curl -v -H "Content-Type: application/json" -X POST --data @isize_job.json "http://<CONTAINER_INSTANCE_IP>:8000/isize"
The content of isize_job.json should contain the job specifics. Some sample job parameters are shown below:
Take the sample input file “/isize/samples/test.mp4” and apply iSIZE “PRECODER_BAL” precoding, resize to 1280×720 and produce output in the shared folder as lossless Y4M file “/isize_files/test_output_resized.y4m” using GPU.
{
"input": "/isize/samples/test.mp4",
"output": "/isize_files/test_output_resized.y4m",
"isize_precoding": "PRECODER_BAL",
"isize_use_gpu": "TRUE",
"codec": "Y4M",
"width": "1280",
"height": "720",
"pix_fmt": "yuv420p10le"
}
Take the sample input file “/isize/samples/test.mp4” and apply iSIZE “PRECODER_LOW” precoding, encode it as AV1 with 2Mbps bitrate and produce output in the shared folder as webm file “/isize_files/test_output_av1.webm” using CPU.
{
"input": "/isize/samples/test.mp4",
"output": "/isize_files/test_output_av1.webm",
"isize_precoding": "PRECODER_LOW",
"isize_use_gpu": "FALSE",
"codec": "AV1",
"bitrate": "2000",
"gop": "60"
}
List current jobs:
curl -X GET http://<CONTAINER_INSTANCE_IP>:8000/isize
Transfer Files
iSIZE container needs direct access to your files before it can process them. You should transfer source media files to the cluster instance before queuing jobs. Similarly, you can get the output files from the instance the same way. You can use any mechanism allowed in AWS to transfer files but SFTP is the most secure and common. A free and popular such SFTP client is FileZilla. Download and install FileZilla and run it. Click “Open Site Manager”.
Click “New Site” and enter a site name. On the right site select and enter the appropriate information. The “<CONTAINER_IP>” needs to be the instance IP in your cluster running the task and “Key file” needs to be the same key that you used when setting up the cluster. Click Connect.
This will connect to the instance. Now you can transfer files to and from the instance+container.
Sample ECS task JSON for GPU instances
The JSON file below can be customized and used to setup and run iSIZE BitSave container in ECS GPU instances. Please make sure you set up appropriate volumes and mount points.
{
"requiresCompatibilities": [
"EC2"
],
"inferenceAccelerators": [],
"containerDefinitions": [
{
"name": "isize_bitsave",
"image": "<CONTAINER_LOCATION>",
"memoryReservation": "20000",
"resourceRequirements": [
{
"type": "GPU",
"value": "1"
}
],
"essential": true,
"portMappings": [
{
"hostPort": "8000",
"containerPort": "8000",
"protocol": "tcp"
},
{
"hostPort": "8001",
"containerPort": "8001",
"protocol": "tcp"
}
],
"environment": null,
"environmentFiles": [],
"secrets": null,
"mountPoints": [
{
"sourceVolume": "isize_shared_folder",
"containerPath": "/isize_files",
"readOnly": ""
}
],
"volumesFrom": null,
"hostname": null,
"user": null,
"workingDirectory": null,
"extraHosts": null,
"logConfiguration": null,
"ulimits": null,
"dockerLabels": null,
"dependsOn": null,
"repositoryCredentials": {
"credentialsParameter": ""
}
}
],
"volumes": [
{
"name": "isize_shared_folder",
"host": {
"sourcePath": "/home/ec2-user"
}
}
],
"networkMode": null,
"memory": "30Gb",
"cpu": null,
"placementConstraints": [],
"family": "isize-task",
"tags": [],
"taskRoleArn": "<IAM_ROLE_CREATED_BEFORE>"
}
Sample ECS task JSON for CPU instances
The JSON file below can be customized and used to setup and run iSIZE BitSave container in ECS CPU instances. Please make sure you set up appropriate volumes and mount points.
{
"requiresCompatibilities": [
"EC2"
],
"inferenceAccelerators": [],
"containerDefinitions": [
{
"name": "isize_bitsave",
"image": "<CONTAINER_LOCATION>",
"memoryReservation": "20000",
"resourceRequirements": null,
"essential": true,
"portMappings": [
{
"hostPort": "8000",
"containerPort": "8000",
"protocol": "tcp"
},
{
"hostPort": "8001",
"containerPort": "8001",
"protocol": "tcp"
}
],
"environment": null,
"environmentFiles": [],
"secrets": null,
"mountPoints": [
{
"sourceVolume": "isize_shared_folder",
"containerPath": "/isize_files",
"readOnly": ""
}
],
"volumesFrom": null,
"hostname": null,
"user": null,
"workingDirectory": null,
"extraHosts": null,
"logConfiguration": null,
"ulimits": null,
"dockerLabels": null,
"dependsOn": null,
"repositoryCredentials": {
"credentialsParameter": ""
}
}
],
"volumes": [
{
"name": "isize_shared_folder",
"host": {
"sourcePath": "/home/ec2-user"
}
}
],
"networkMode": null,
"memory": "30Gb",
"cpu": null,
"placementConstraints": [],
"family": "isize-task",
"tags": [],
"taskRoleArn": "<IAM_ROLE_CREATED_BEFORE>"
}
Recommended Instances
The following instance types are recommended:
m6g.8xlarge
m6g.12xlarge
m6gd.8xlarge
m6gd.12xlarge
m5.8xlarge
m5.12xlarge
m5d.8xlarge
m5d.12xlarge
c6g.8xlarge
c6g.12xlarge
c5.4xlarge
c5.9xlarge
c5.12xlarge
c5ad.4xlarge
c5ad.8xlarge
c5ad.12xlarge
c5d.4xlarge
c5d.9xlarge (recommended CPU instance)
c5d.12xlarge
c4.8xlarge
g4dn.4xlarge
g4dn.8xlarge (recommended GPU instance)
g4dn.12xlarge
g3.8xlarge
g3.16xlarge
r5.8xlarge
r5.12xlarge
r5a.8xlarge
r5a.12xlarge
r5d.8xlarge
r5d.12xlarge
r5dn.4xlarge
r5dn.8xlarge
r4.8xlarge
r4.16xlarge
For optimal performance, we recommend “g4dn.8xlarge” instance for GPU mode and “c5d.9xlarge” instance for CPU mode.
Error Handling
The iSIZE BitSave container runs within the ECS cluster in an instance. After you queue the job, depending on the type of instance chosen it will take some time for the output to appear. You can check the progress of the job by sending either “list jobs” or “list job” REST requests. If the job status shows an error then it will list the reason for the error. In most cases, the reason for the error is an unsupported codec. Check the supported codec in the “add job” section of this manual. If a supported codec file does not produce output then retry with the same job first. The container will accept new jobs as long as the cluster task is running and the EC2 instance is alive. If the container does not respond even though the instance is running, then stop the task and then run the task again so that the container is refreshed. If you still face issues please contact iSIZE support at support@isize.co . If you created the cluster properly and started the task but an EC2 instance is not started then it could be due to the unavailability of that instance type in that region or due to your service limit. You should try again later or if the issue persists then contact AWS support.
Subscription
You can always view and manage your AWS marketplace from the marketplace subscription page. Please be aware that you will still be billed for any running task even if you delete the subscriptions. So please stop and delete any task to avoid being billed.
Technical Support
If you need additional assistance please contact support@isize.co
