my-asu-archive/CSE546_Cloud_Computing_Projects
1
0
Fork 0
mirror of https://github.com/20kaushik02/CSE546_Cloud_Computing_Projects.git synced 2026-01-25 06:44:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

did not do part 2. but must complete even if of no use.

Browse Source
This commit is contained in:
Kaushik Narayan R
2024-03-17 22:16:52 -07:00
parent 56c521ae2e
commit 582233502d
1273 changed files with 9072 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

217
Tooling/scripts/README.md Normal file
View File

@@ -0,0 +1,217 @@
### Grading Scripts
- [Project-1 Part-1 Test Case 1](https://github.com/visa-lab/CSE546-Cloud-Computing/blob/main/scripts/project1_grader.py)
- Test Objective:
| Test # | Test Case | Test Criteria | Poor | Good | Excellent | Total Points |
|--------|----------------------|------------------------------------------------------------------------------------------------------------------------------|------------------------------------------|-----------------------------------------------------|----------------------------------------------------------|--------------|
| 1 | Validate EC2 Instance | "To check if 1) If there exists a EC2 instance with name ""web-instance"" 2) if exists then check if the state of the web-instance in ""running""" | There is no EC2 intance with name "web-instance" (0) | The EC2 instance with the name "web-instance" exists; but is not in "running" state (5) | The EC2 instance with the name "web-instance" exists; and is in "running" state (10) | 10 |
- How to run the script:
```
usage: project1_grader.py [-h] [--access_keyId ACCESS_KEYID] [--access_key ACCESS_KEY]
Grading Script
options:
-h, --help show this help message and exit
--access_keyId ACCESS_KEYID ACCCESS KEY ID of the grading IAM user
--access_key ACCESS_KEY SECRET ACCCESS KEY of the grading IAM user
```
- [Project-1 Part-2 Grading Script](https://github.com/visa-lab/CSE546-Cloud-Computing/blob/main/scripts/p2_grader.py):
- How to use the script:
```
usage: p2_grader.py [-h] [--access_keyId ACCESS_KEYID] [--access_key ACCESS_KEY] [--req_sqs REQ_SQS] [--resp_sqs RESP_SQS] [--in_bucket IN_BUCKET] [--out_bucket OUT_BUCKET]
Grading Script
options:
-h, --help show this help message and exit
--access_keyId ACCESS_KEYID ACCCESS KEY ID of the grading IAM user
--access_key ACCESS_KEY SECRET ACCCESS KEY of the grading IAM user
--req_sqs REQ_SQS Name of the Request SQS Queue
--resp_sqs RESP_SQS Name of the Response SQS Queue
--in_bucket IN_BUCKET Name of the S3 Input Bucket
--out_bucket OUT_BUCKET Name of the S3 Output Bucket
```
**Note**: We will follow the naming conventions for S3 Bucket and SQS Queue names as described in the project document to grade your submission
```
=============================================================================
======== Welcome to CSE546 Cloud Computing AWS Console ======================
=============================================================================
IAM ACESS KEY ID: XXXXXXXXXXXXX
IAM SECRET ACCESS KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=============================================================================
1 - Validate EC2 Instances
2 - Validate S3 Buckets
3 - Validate SQS Queues
4 - Validate autoscaling
0 - Exit
Enter a choice:
```
- **Validate EC2 Instance**:
```
=============================================================================
======== Welcome to CSE546 Cloud Computing AWS Console ======================
=============================================================================
IAM ACESS KEY ID: XXXXXXXXXXXXX
IAM SECRET ACCESS KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=============================================================================
1 - Validate EC2 Instances
2 - Validate S3 Buckets
3 - Validate SQS Queues
4 - Validate autoscaling
0 - Exit
Enter a choice:
1
Found 1 web-tier instances in running state.
Found 0 app-tier instanes in running state
```
- **Validate S3 Buckets**
```
=============================================================================
======== Welcome to CSE546 Cloud Computing AWS Console ======================
=============================================================================
IAM ACESS KEY ID: XXXXXXXXXXXXX
IAM SECRET ACCESS KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=============================================================================
1 - Validate EC2 Instances
2 - Validate S3 Buckets
3 - Validate SQS Queues
4 - Validate autoscaling
0 - Exit
Enter a choice:
2
- WARN: If there are objects in the S3 buckets; they will be deleted
---------------------------------------------------------
S3 Input Bucket:12345678910-in-bucket has 0 object(s)
S3 Output Bucket:12345678910-out-bucket has 0 object(s)
```
- **Validate SQS Queue**
```
=============================================================================
======== Welcome to CSE546 Cloud Computing AWS Console ======================
=============================================================================
IAM ACESS KEY ID: XXXXXXXXXXXXX
IAM SECRET ACCESS KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=============================================================================
1 - Validate EC2 Instances
2 - Validate S3 Buckets
3 - Validate SQS Queues
4 - Validate autoscaling
0 - Exit
Enter a choice:
3
- The expectation is the both the Request and Response SQS should exist and be EMPTY
- WARN: This will purge any messages available in the SQS
---------------------------------------------------------
SQS Request Queue:12345678910-req-queue has 0 pending messages.
SQS Response Queue:12345678910-resp-queue has 0 pending messages.
```
- **Validate autoscaling**
```
=============================================================================
======== Welcome to CSE546 Cloud Computing AWS Console ======================
=============================================================================
IAM ACESS KEY ID: XXXXXXXXXXXXX
IAM SECRET ACCESS KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
=============================================================================
1 - Validate EC2 Instances
2 - Validate S3 Buckets
3 - Validate SQS Queues
4 - Validate autoscaling
0 - Exit
Enter a choice:
4
- Run this BEFORE the workload generator client starts. Press Ctrl^C to exit.
- The expectation is as follows:
-- # of app tier instances should gradually scale and eventually reduce back to 0
-- # of SQS messages should gradually increase and eventually reduce back to 0
------------------------------------------------------------------------------------------------------------------
| # of messages in | # of messages in | # of EC2 instances | # of objects in S3 | # of objects in S3 |
| SQS Request Queue | SQS Response Queue | in running state | Input Bucket | Output Bucket |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 0 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 1 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 2 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 4 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 8 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 11 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 16 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 17 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 19 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 50 | 0 | 20 | 0 | 0 |
------------------------------------------------------------------------------------------------------------------
| 48 | 0 | 20 | 2 | 2 |
------------------------------------------------------------------------------------------------------------------
| 35 | 0 | 20 | 2 | 2 |
------------------------------------------------------------------------------------------------------------------
| 35 | 0 | 20 | 9 | 9 |
------------------------------------------------------------------------------------------------------------------
| 35 | 0 | 20 | 12 | 12 |
------------------------------------------------------------------------------------------------------------------
| 32 | 0 | 20 | 21 | 21 |
------------------------------------------------------------------------------------------------------------------
| 42 | 4 | 20 | 28 | 29 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 40 | 40 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 47 | 47 |
------------------------------------------------------------------------------------------------------------------
| 4 | 0 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 41 | 4 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 3 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 12 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 8 | 20 | 49 | 49 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 3 | 20 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 20 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 15 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 6 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
| 0 | 0 | 0 | 50 | 50 |
------------------------------------------------------------------------------------------------------------------
```
The above example output is for a 50-requests workload.
**Note**: This does not show all the test cases for the project. Please refer to the project document for more details.

202
Tooling/scripts/p2_grader.py Normal file
View File

@@ -0,0 +1,202 @@
__copyright__ = "Copyright 2024, VISA Lab"
__license__ = "MIT"
import os
import pdb
import time
import json
import boto3
import httpx
import sys
import argparse
import textwrap
class aws_grader():
def __init__(self, access_keyId, access_key, req_sqs, resp_sqs, in_bucket, out_bucket):
self.iam_access_keyId = access_keyId
self.iam_secret_access_key = access_key
self.iam_session = boto3.Session(aws_access_key_id = self.iam_access_keyId,
aws_secret_access_key = self.iam_secret_access_key)
self.ec2_resources = self.iam_session.resource('ec2', 'us-east-1')
self.s3_resources = self.iam_session.resource('s3', 'us-east-1')
self.sqs_resources = self.iam_session.resource('sqs', 'us-east-1')
self.sqs_client = self.iam_session.client('sqs', 'us-east-1')
self.req_sqs_name = req_sqs
self.resp_sqs_name = resp_sqs
self.in_bucket_name = in_bucket
self.out_bucket_name = out_bucket
self.app_tier_tag = "app-tier-instance"
self.web_tier_tag = "web-instance"
def get_instance_details(self, tag, state):
instances = self.ec2_resources.instances.filter(
Filters=[
{'Name': 'tag:Name', 'Values': [tag+"*"]},
{'Name': 'instance-state-name', 'Values': [state]}
]
)
return len(list(instances))
def validate_ec2_instance(self):
web_instances = self.get_instance_details(self.web_tier_tag, 'running')
app_instances = self.get_instance_details(self.app_tier_tag, 'running')
print(f"Found {web_instances} web-tier instances in running state.")
print(f"Found {app_instances} app-tier instanes in running state")
def empty_s3_bucket(self, bucket_name):
bucket = self.s3_resources.Bucket(bucket_name)
bucket.objects.all().delete()
print(f"{bucket_name} S3 Bucket is now EMPTY !!")
def count_bucket_objects(self, bucket_name):
bucket = self.s3_resources.Bucket(bucket_name)
count = 0
for index in bucket.objects.all():
count += 1
#print(f"{bucket_name} S3 Bucket has {count} objects !!")
return count
def validate_s3_buckets(self):
print(" - WARN: If there are objects in the S3 buckets; they will be deleted")
print(" ---------------------------------------------------------")
ip_obj_count = self.count_bucket_objects(self.in_bucket_name)
op_obj_count = self.count_bucket_objects(self.out_bucket_name)
print(f"S3 Input Bucket:{self.in_bucket_name} has {ip_obj_count} object(s)")
print(f"S3 Output Bucket:{self.out_bucket_name} has {op_obj_count} object(s)")
if ip_obj_count:
self.empty_s3_bucket(self.in_bucket_name)
if op_obj_count:
self.empty_s3_bucket(self.out_bucket_name)
def get_sqs_queue_length(self, sqs_queue_name):
num_requests = self.sqs_client.get_queue_attributes(
QueueUrl=sqs_queue_name,
AttributeNames=['ApproximateNumberOfMessages'])
return int(num_requests['Attributes']['ApproximateNumberOfMessages'])
def validate_sqs_queues(self):
print(" - The expectation is the both the Request and Response SQS should exist and be EMPTY")
print(" - WARN: This will purge any messages available in the SQS")
print(" ---------------------------------------------------------")
try:
req_sqs = self.sqs_resources.get_queue_by_name(QueueName=self.req_sqs_name)
resp_sqs = self.sqs_resources.get_queue_by_name(QueueName=self.resp_sqs_name)
ip_queue_requests = self.get_sqs_queue_length(self.req_sqs_name)
op_queue_response = self.get_sqs_queue_length(self.resp_sqs_name)
print(f"SQS Request Queue:{self.req_sqs_name} has {ip_queue_requests} pending messages.")
print(f"SQS Response Queue:{self.resp_sqs_name} has {op_queue_response} pending messages.")
if ip_queue_requests:
print(" - WARN: Purging the Requeust SQS. Waiting 60 seconds ..")
self.sqs_client.purge_queue(QueueUrl=self.req_sqs_name)
time.sleep(60)
if op_queue_response:
print(" - WARN: Purging the SQS. Waiitng 60 seconds ..")
self.sqs_client.purge_queue(QueueUrl=self.resp_sqs_name)
time.sleep(60)
except Exception as ex:
print(f"SQS Queues Error: {ex}. Please Check your AWS Account")
def beautify_headers(self):
column1 = " # of messages in SQS Request Queue "
column2 = " # of messages in SQS Response Queue "
column3 = " # of EC2 instances in running state "
column4 = " # of objects in S3 Input Bucket "
column5 = " # of objects in S3 Output Buket "
column_width = 20
wrapped_column1 = textwrap.fill(column1, column_width)
wrapped_column2 = textwrap.fill(column2, column_width)
wrapped_column3 = textwrap.fill(column3, column_width)
wrapped_column4 = textwrap.fill(column4, column_width)
wrapped_column5 = textwrap.fill(column5, column_width)
lines1 = wrapped_column1.split('\n')
lines2 = wrapped_column2.split('\n')
lines3 = wrapped_column3.split('\n')
lines4 = wrapped_column4.split('\n')
lines5 = wrapped_column5.split('\n')
print("-" * 114)
for line1, line2, line3, line4, line5 in zip(lines1, lines2, lines3, lines4, lines5):
print(f"| {line1.center(column_width)} | {line2.center(column_width)} | {line3.center(column_width)} | {line4.center(column_width)} | {line5.center(column_width)} |")
print("-" *114)
def validate_autoscaling(self):
print(" - Run this BEFORE the workload generator client starts. Press Ctrl^C to exit.")
print(" - The expectation is as follows:")
print(" -- # of app tier instances should gradually scale and eventually reduce back to 0")
print(" -- # of SQS messages should gradually increase and eventually reduce back to 0")
self.beautify_headers()
format_string = "| {:^20} | {:^20} | {:^20} | {:^20} | {:^20} |"
while True:
req_queue_count = self.get_sqs_queue_length(self.req_sqs_name)
resp_queue_count = self.get_sqs_queue_length(self.resp_sqs_name)
num_instances = self.get_instance_details(self.app_tier_tag, 'running')
ip_obj_count = self.count_bucket_objects(self.in_bucket_name)
op_obj_count = self.count_bucket_objects(self.out_bucket_name)
print(format_string.format(req_queue_count, resp_queue_count, num_instances, ip_obj_count, op_obj_count))
print("-" * 114)
time.sleep(2)
def display_menu(self):
print("\n")
print("=============================================================================")
print("======== Welcome to CSE546 Cloud Computing AWS Console ======================")
print("=============================================================================")
print(f"IAM ACESS KEY ID: {self.iam_access_keyId}")
print(f"IAM SECRET ACCESS KEY: {self.iam_secret_access_key}")
print("=============================================================================")
print("1 - Validate EC2 Instances")
print("2 - Validate S3 Buckets")
print("3 - Validate SQS Queues")
print("4 - Validate autoscaling")
print("0 - Exit")
print("Enter a choice:")
choice = input()
return choice
def main(self):
while(1):
choice = self.display_menu()
if int(choice) == 1:
self.validate_ec2_instance()
elif int(choice) == 2:
self.validate_s3_buckets()
elif int(choice) == 3:
self.validate_sqs_queues()
elif int(choice) == 4:
self.validate_autoscaling()
elif int(choice) == 0:
break
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Grading Script')
parser.add_argument('--access_keyId', type=str, help='ACCCESS KEY ID of the grading IAM user')
parser.add_argument('--access_key', type=str, help='SECRET ACCCESS KEY of the grading IAM user')
parser.add_argument('--req_sqs', type=str, help="Name of the Request SQS Queue")
parser.add_argument('--resp_sqs', type=str, help="Name of the Response SQS Queue")
parser.add_argument('--in_bucket', type=str, help='Name of the S3 Input Bucket')
parser.add_argument('--out_bucket', type=str, help='Name of the S3 Output Bucket')
args = parser.parse_args()
access_keyId = args.access_keyId
access_key = args.access_key
req_sqs = args.req_sqs
resp_sqs = args.resp_sqs
in_bucket = args.in_bucket
out_bucket = args.out_bucket
aws_obj = aws_grader(access_keyId, access_key, req_sqs, resp_sqs, in_bucket, out_bucket)
aws_obj.main()

78
Tooling/scripts/project1_grader.py Normal file
View File

@@ -0,0 +1,78 @@
__copyright__ = "Copyright 2024, VISA Lab"
__license__ = "MIT"
import os
import boto3
import dotenv
import httpx
import pdb
import json
import argparse
class aws_grader():
def __init__(self, access_keyId, access_key):
self.iam_access_keyId = access_keyId
self.iam_secret_access_key = access_key
self.iam_session = boto3.Session(aws_access_key_id = self.iam_access_keyId,
aws_secret_access_key = self.iam_secret_access_key)
self.ec2_resources = self.iam_session.resource('ec2', 'us-east-1')
def get_tag(self, tags, key='Name'):
if not tags:
return 'None'
for tag in tags:
if tag['Key'] == key:
return tag['Value']
return 'None'
'''
Test Case:1
1) Checks if there is a web EC2 instance with the name "web-instance"
2) Checks if the state of "web-instance" is "running".
'''
def test_case_1(self):
message = ""
for instance in self.ec2_resources.instances.all():
name = self.get_tag(instance.tags)
if name == "web-instance":
message += "web-tier instance found."
state = instance.state['Name']
message += f" With state: {state}"
if state == "running":
self.web_tier_instanceId = instance.id
print(f"Test Case:1 Passed. {message}")
return
else:
print(f"Test Case:1 Failed. {message}")
return
message += "web-tier Instance Not Found."
print("Test Case:1 Failed. {message}")
def main(self):
# Use the session to access resources via the role
print("======== Welcome to CSE546 Cloud Computing Grading Console ==================")
print(f"IAM ACESS KEY ID: {self.iam_access_keyId}")
print(f"IAM SECRET ACCESS KEY: {self.iam_secret_access_key}")
self.test_case_1()
if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Upload images')
parser.add_argument('--access_keyId', type=str, help='ACCCESS KEY ID of the grading IAM user')
parser.add_argument('--access_key', type=str, help='SECRET ACCCESS KEY of the grading IAM user')
args = parser.parse_args()
access_keyId = args.access_keyId
access_key = args.access_key
aws_obj = aws_grader(access_keyId, access_key)
aws_obj.main()