Управление кластером Spark ---  --- ### Управление конфигурациями, IaC    --- ### Плюсы минусы камни --- ### Плюсы минусы камни * Плюсы * Быстрый старт * Универсальность * Минусы * Много низкоуровневых деталей * Есть состояние * Неочевидная изоляция --- ### Осталось разобраться с одной маленькой проблемой --- ### Dependency Hell  ---  --- https://github.com/bigartm/bigartm   ---   In June, 2003, a successful 100-million-page demonstration system was developed. To meet the multi-machine processing needs of the crawl and index tasks, the Nutch project has also implemented a MapReduce facility and a distributed file system. The two facilities have been spun out into their own subproject, called Hadoop. --- Spark - очень удачно взлетевший PoC для Mesos  --- {%pdf https://people.eecs.berkeley.edu/~alig/papers/mesos.pdf %} ---  --- {%pdf https://cs.stanford.edu/~matei/papers/2011/nsdi_drf.pdf %} ---  ---  --- {%pdf https://www.usenix.org/legacy/event/atc10/tech/full_papers/Hunt.pdf %} --- {%pdf https://fpjdotme.files.wordpress.com/2016/07/yl-2010-007.pdf %} ---  ---