---
# System prepended metadata

title: Exercise 8

---

310088 295694 297138 314706 311844

Update this file with a copy of your group's solution.
Solution:

# Exercise 8

## Question 1.1

The bad performance comme from the fact that the data are hash partitionned and that force the node to exchange a lot of data resulting in time loss.

## Question 1.2
```scala=
people.groupBy(_.age).map {
  case (age, peopleOfAge) =>
    val salaries = peopleOfAge.map(_.salary)
    val n = peopleOfAge.size
    val mean = salaries.sum / n 
    val variance = salaries.map(x => Math.pow(x,2)).sum / n - Math.pow(mean,2)

    (age, (mean, Math.sqrt(variance).toLong))
}
```

## Question 1.3
```scala =
val pairs = people.map(p => (p.age, p.salary))
val partitionner = new RangePartitioner(32, pairs)
val partition = pairs.partitionBy(partitionner).persist()

partition.groupByKey().map((age, iterableOfSalary) => {
    val n = iterableOfSalary.size
    val mean = iterableOfSalary.sum / n 
    val variance = iterableOfSalary.map(x => Math.pow(x,2)).sum / n - Math.pow(mean,2)

    (age, (mean, Math.sqrt(variance).toLong))
})
```
### Question 2.1
A partitioner is an object representing a partition function, i.e. an object that can tell, given an element of an RDD, which partition this element should be part of.
Repartitioning data can be useful to:
    - allow analyzing big databases by storing data on several computers
    - group elements that should be processed together to minimize network traffic

### Question 2.2
`mapValues`, `filter`, `flatMapValues`, `reduceByKey` and `groupByKey` preserve the partitioner of their parent because the RDD produced by the transformation has the same keys as the initial RDD.

### Question 2.3
`join`, `reduceByKey` and `groupByKey`, because they may have to consider data from different nodes and therefore have to repartition the new RDD they created instead of just keeping data of the new collection on the same node as their parent partition, like other functions do.

### Question 2.4
A `HashPartitioner` maps an element to its partition by computing its hash value modulo the number of partitions
A `RangePartitioner` uses an ordering and maps an element to its partition by computing the interval the element belongs to
A `HashPartitioner` can be useful when the data has no natural ordering or when the ordering does not matter for the analysis. A `RangePartitioner` can be useful when the ordering matters for analysis, or if for a specific reason hashing leads to an unbalanced partition of the data.
