---
# System prepended metadata

title: Untitled

---

## A simple spark program for you to try

1. Edge nodes / S3 bucket - daily files are coming 
2. File format - XYZ_20221021.zip
3. Inside Zip - you have employee.csv file with columns - first name, last name, id, salary, dept_id
  
### Transformations needed 
- create a new name column -  name =  first name + last name
- calculate salary ranking for each employee within dept_id and add a new column - salary ranking. E.g., - for IT or Admin dept if employee salary is at the highest level or 2nd hihest etc.
- Ingest final data to Hive table/ HDFS or S3 bucket - partitioned date from file name itself "20221021"
  
  
  
 ```
 ### put your code here
 ```
path='../employee.csv'
df = pd.read_csv(path)

df['name'] = df.apply(lambda x: x['first_name'] + x['last_name'])
  
df_grps = df.groupby(['dept_id'])
for indx,group in df_grps:
    group.sort_values(key='salary')
    ranks = list(range(1, len(group) + 1))
    group['ranks'] = pd.Series(ranks)